2024-03-25T17:53:24.039 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:53:24.039 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/orig.config.yaml 2024-03-25T17:53:24.082 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:53:24.171 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621318' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDRDJbZDpFbLIzdGv/r9q9qfwyAtOCD9AXuxprSspTsrYGvfwm7W3TqHygDf1BCqYWLQ3OzFfScuo2cPStyYEQ= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBTCwSZrmGLv4xlFHncf1n8NvCYQh1zPjWy/plhZ2/6WW1z/UzXC2gWa2V0i5EyKIgN62bv3j0SjgFo+YQo7vQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:53:24.171 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:53:24.172 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:53:24.172 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:53:24.173 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:53:24.197 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:53:24.198 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:53:24.198 INFO:teuthology.packaging:ref: None 2024-03-25T17:53:24.198 INFO:teuthology.packaging:tag: None 2024-03-25T17:53:24.198 INFO:teuthology.packaging:branch: main 2024-03-25T17:53:24.198 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:53:24.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:53:24.547 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:53:24.549 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:53:24.560 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:53:24.560 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:53:24.574 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:53:24.590 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:53:24.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:53:24.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:48:05.211334', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDRDJbZDpFbLIzdGv/r9q9qfwyAtOCD9AXuxprSspTsrYGvfwm7W3TqHygDf1BCqYWLQ3OzFfScuo2cPStyYEQ='} 2024-03-25T17:53:24.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:48:05.209645', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBTCwSZrmGLv4xlFHncf1n8NvCYQh1zPjWy/plhZ2/6WW1z/UzXC2gWa2V0i5EyKIgN62bv3j0SjgFo+YQo7vQ='} 2024-03-25T17:53:24.647 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:53:24.654 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:53:24.654 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:53:24.654 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:53:24.732 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fecfd1bd0d0>, signals=[15]) 2024-03-25T17:53:24.732 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:53:24.740 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:53:24.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T17:53:24.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:24.849 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T17:53:24.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:24.935 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:53:24.945 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-25T17:53:24.962 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-25T17:53:24.963 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-03-25T17:53:25.020 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:53:25.021 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:53:25.021 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-25T17:53:25.044 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-25T17:53:25.062 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-25T17:53:25.062 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-25T17:53:25.117 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:53:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:53:25.119 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-25T17:53:25.144 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:53:25.155 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:53:25.167 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:53:25.168 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:53:25.170 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:53:25.186 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:53:25.198 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:53:25.198 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:53:25.228 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:53:25.262 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:53:25.314 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:53:25.330 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:53:25.330 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:53:25.330 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:53:25.330 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:53:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:53:25.347 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-25T17:53:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:53:25.347 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-25T17:53:25.402 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:53:25.402 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-03-25T17:53:25.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-03-25T17:53:25.406 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:53:25.406 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-03-25T17:53:25.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-03-25T17:53:27.747 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-03-25T17:53:28.527 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T17:53:29.139 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T17:53:29.986 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-25T17:54:08.981 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 208 kB/s | 8.0 MB 00:39 2024-03-25T17:54:09.764 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 209 kB/s | 8.0 MB 00:39 2024-03-25T17:54:17.593 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-25T17:54:18.859 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-25T17:54:28.522 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 851 kB/s | 6.1 MB 00:07 2024-03-25T17:54:29.660 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 871 kB/s | 6.1 MB 00:07 2024-03-25T17:54:29.900 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-25T17:54:30.802 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-25T17:54:32.158 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T17:54:33.945 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T17:54:35.785 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T17:54:37.917 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:54:37.917 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:54:38.008 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T17:54:38.018 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T17:54:38.018 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-25T17:54:38.018 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T17:54:38.018 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-03-25T17:54:38.019 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:54:38.020 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 76 M 2024-03-25T17:54:38.020 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-03-25T17:54:38.020 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T17:54:38.186 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:38.187 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:38.187 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:38.252 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.253 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.253 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.404 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.415 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.426 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.432 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.482 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:38.490 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.503 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.503 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.535 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.554 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:38.555 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.602 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.643 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.669 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.669 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:38.805 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.806 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.824 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.825 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.826 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.846 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:38.984 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:38.985 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:38.996 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T17:54:39.005 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:39.335 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:39.356 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:39.378 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:39.399 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.421 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.479 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.536 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.544 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.601 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:39.611 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.620 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.696 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.759 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.770 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.804 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:39.811 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:39.874 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:39.898 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:39.898 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:39.920 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:39.920 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:40.370 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.392 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.417 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.694 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.712 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.729 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.743 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:40.745 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.793 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.812 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.813 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.829 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:40.833 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:40.834 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:40.892 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:40.928 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:40.947 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:40.966 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:41.081 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.094 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.170 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.180 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.182 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.196 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:54:41.196 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:54:41.257 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.261 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:41.296 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.298 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T17:54:41.313 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-03-25T17:54:41.314 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:54:41.315 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 76 M 2024-03-25T17:54:41.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-03-25T17:54:41.318 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T17:54:41.319 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.351 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.352 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.369 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.387 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:41.387 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:41.388 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:41.544 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 20 kB/s | 66 kB 00:03 2024-03-25T17:54:41.623 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.633 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.641 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:41.669 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.670 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.675 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:41.686 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.692 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:41.794 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:41.802 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:41.833 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:41.965 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:41.970 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:41.980 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:42.007 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.011 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.021 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.8 MB/s | 15 MB 00:03 2024-03-25T17:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:42.123 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.123 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.132 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.155 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.191 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.328 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.329 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.361 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.679 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.711 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.3 MB/s | 33 MB 00:04 2024-03-25T17:54:42.711 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:42.713 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.749 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.749 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.784 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.784 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.800 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:42.802 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.802 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.819 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:42.851 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:42.865 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:42.933 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:42.962 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:42.985 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:42.993 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:43.011 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.047 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:43.049 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:43.065 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.065 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.098 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:43.117 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.118 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.172 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:43.399 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:43.475 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:43.572 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:43.597 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:43.600 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:43.649 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:43.696 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:43.795 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:43.894 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:43.920 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:43.970 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:43.999 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:44.036 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:44.100 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:44.116 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:44.138 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:44.182 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.202 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.202 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.277 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:44.320 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.329 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.344 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.350 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.367 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:44.401 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.402 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:44.424 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.428 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.459 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.539 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.571 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.602 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.659 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.678 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.691 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.696 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.725 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:44.739 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.745 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:44.783 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.799 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.801 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.822 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.836 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.837 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:44.981 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:44.990 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 18 kB/s | 66 kB 00:03 2024-03-25T17:54:45.052 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:45.106 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:45.182 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:45.216 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:45.334 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:45.389 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:45.430 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:46.066 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.1 MB/s | 33 MB 00:04 2024-03-25T17:54:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:46.144 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.0 MB/s | 27 MB 00:04 2024-03-25T17:54:46.151 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:46.151 INFO:teuthology.orchestra.run.smithi196.stdout:Total 9.3 MB/s | 76 MB 00:08 2024-03-25T17:54:46.408 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.0 MB/s | 15 MB 00:05 2024-03-25T17:54:46.489 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:46.582 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:46.639 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:46.816 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T17:54:46.993 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:47.077 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:47.094 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:47.199 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T17:54:47.199 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T17:54:47.225 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:47.769 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:47.856 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:48.207 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T17:54:48.207 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T17:54:48.268 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:48.444 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:48.472 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:48.599 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:48.764 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:48.847 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:48.882 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:49.725 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.8 MB/s | 27 MB 00:04 2024-03-25T17:54:49.732 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:49.732 INFO:teuthology.orchestra.run.smithi029.stdout:Total 9.0 MB/s | 76 MB 00:08 2024-03-25T17:54:50.427 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T17:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T17:54:50.881 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T17:54:50.881 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T17:54:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:54:51.470 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:54:51.904 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T17:54:51.904 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T17:54:54.252 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:54:54.379 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:54:54.510 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T17:54:55.212 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:54:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:54:58.272 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:54:58.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:54:59.083 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:54:59.170 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:55:03.597 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:03.976 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:33.320 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:55:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:55:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:33.741 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:33.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:55:33.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:55:33.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:55:34.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:34.178 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:55:34.178 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T17:55:34.178 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.178 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.179 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.179 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.179 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:55:34.179 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T17:55:34.506 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:55:34.603 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.604 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:55:34.604 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:55:34.604 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:55:34.604 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:55:34.604 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:55:34.604 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:55:34.605 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:55:34.684 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:34.684 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:55:34.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-03-25T17:55:34.718 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-03-25T17:55:34.718 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-03-25T17:55:34.718 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-03-25T17:55:34.718 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-03-25T17:55:34.718 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-03-25T17:55:34.719 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T17:55:34.720 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:55:34.721 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:55:34.721 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:55:34.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:55:35.340 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-03-25T17:55:36.722 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:55:36.758 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-03-25T17:55:36.760 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-25T17:55:36.776 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.wu61WIey7J 2024-03-25T17:55:36.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wu61WIey7J 2024-03-25T17:55:36.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.wu61WIey7J 2024-03-25T17:55:36.982 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.wu61WIey7J is 6KB 2024-03-25T17:55:37.032 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.wu61WIey7J 2024-03-25T17:55:37.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:55:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:55:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:55:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:55:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:55:37.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:55:37.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-03-25T17:55:41.652 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:41.652 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:55:41.652 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:55:41.698 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:42.154 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:42.155 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:55:42.155 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:55:42.155 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:55:42.603 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:55:42.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:55:42.603 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T17:55:42.603 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:42.603 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:55:42.604 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:55:42.604 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:55:42.604 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:55:42.604 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T17:55:43.027 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:55:43.169 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:43.169 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:55:43.169 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:55:43.169 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:55:43.169 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:55:43.169 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:55:43.169 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:55:43.169 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:55:43.253 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:55:43.253 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:55:43.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-03-25T17:55:43.308 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-03-25T17:55:43.308 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-03-25T17:55:43.308 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:55:43.309 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:55:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:55:43.311 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-03-25T17:55:43.311 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T17:55:43.311 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:55:43.312 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:55:43.312 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:55:43.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:55:44.003 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-03-25T17:55:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:55:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-03-25T17:55:45.675 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-25T17:55:45.690 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.dwwFZTtQoF 2024-03-25T17:55:45.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dwwFZTtQoF 2024-03-25T17:55:45.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.dwwFZTtQoF 2024-03-25T17:55:45.912 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.dwwFZTtQoF is 6KB 2024-03-25T17:55:45.959 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.dwwFZTtQoF 2024-03-25T17:55:45.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:55:46.046 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:55:46.047 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:55:46.047 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:55:46.047 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:55:46.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:55:46.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-03-25T17:56:07.278 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:56:07.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-25T17:56:07.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:56:16.222 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:56:16.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-25T17:56:16.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:57:07.331 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:57:16.234 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:57:16.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-25T17:57:16.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:57:16.732 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-25T17:57:17.094 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-25T17:57:17.094 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:57:17.094 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-25T17:57:17.107 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:57:17.108 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:57:17.108 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:57:17.108 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:57:18.110 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:57:18.110 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:57:18.192 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-03-25T17:57:18.237 DEBUG:teuthology.parallel:result is None 2024-03-25T17:57:25.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-25T17:57:25.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:57:29.240 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-25T17:57:30.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-25T17:57:30.275 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:57:30.275 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-25T17:57:30.289 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:57:30.289 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:57:30.289 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:57:30.289 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:57:31.290 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:57:31.291 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:57:31.444 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-03-25T17:57:31.507 DEBUG:teuthology.parallel:result is None 2024-03-25T17:57:31.508 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:57:31.517 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:57:31.517 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:57:31.520 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:57:31.538 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:57:31.550 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:57:31.564 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:57:31.564 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:57:31.574 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:57:31.623 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:57:31.631 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:57:31.631 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:57:31.649 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:57:31.695 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:57:31.698 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:57:31.709 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:57:31.711 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:57:31.713 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:57:31.722 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:57:31.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:57:31.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:57:31.780 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:57:31.791 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:57:31.792 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:57:31.794 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:57:31.835 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:57:31.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:57:31.934 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:57:32.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:57:32.057 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T17:57:32.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:57:32.142 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:57:32.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:57:32.246 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:57:32.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:57:32.333 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T17:57:32.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:57:32.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-03-25T17:57:32.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-25T17:57:32.445 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:57:32.494 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:57:32.754 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:57:32.764 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:57:32.764 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:57:32.780 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:57:32.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-03-25T17:57:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-03-25T17:57:32.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-25T17:57:32.963 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-25T17:57:32.965 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:57:32.965 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-03-25T17:57:32.991 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-03-25T17:57:32.991 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-25T17:57:33.022 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-25T17:57:33.023 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:57:33.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:57:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:57:33.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:57:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:57:33.091 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:57:33.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:57:33.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:57:33.215 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:57:33.226 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T17:57:33.231 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T17:57:33.359 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:57:33.374 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:57:33.375 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:02:03.659 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-25T18:02:03.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-25T18:02:03.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:02:03.757 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-25T18:02:03.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-25T18:02:03.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-25T18:02:03.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:02:03.918 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-25T18:02:03.997 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-25T18:02:03.997 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:02:04.008 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:02:04.008 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:02:04.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:02:04.011 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:02:04.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:02:04.040 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:02:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:02:04.068 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:02:04.084 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:02:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-03-25T18:02:04.100 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-03-25T18:02:04.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-25T18:02:04.116 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:02:04.126 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-25T18:02:04.132 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:02:04.143 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:02:04.159 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:02:04.186 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:02:04.189 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:02:04.189 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-25T18:02:04.190 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.190 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.190 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.190 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.220 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:02:04.224 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:02:04.232 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:02:04.232 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:02:04.232 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:02:04.232 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:02:04.238 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:02:04.238 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:02:04.238 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:02:04.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:02:04.241 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:02:04.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:02:04.381 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:02:04.382 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:02:04.389 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:02:04.389 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:02:04.786 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:02:04.786 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T18:02:04.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:02:04.792 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:02:04.792 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T18:02:04.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:02:04.816 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:02:04.817 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:02:04.820 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:02:04.820 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:02:04.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:02:04.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:02:04.968 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:02:04.969 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:02:05.035 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-03-25T18:02:05.035 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-25T18:02:05.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-03-25T18:02:05.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-25T18:02:05.283 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-03-25T18:02:05.307 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-25T18:02:05.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-03-25T18:02:05.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-03-25T18:02:06.529 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 92 kB/s | 82 kB 00:00 2024-03-25T18:02:06.747 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 72 kB/s | 82 kB 00:01 2024-03-25T18:02:07.313 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T18:02:07.648 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:02:08.146 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:02:08.516 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T18:02:08.624 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:02:09.046 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:02:10.299 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T18:02:10.555 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-25T18:02:18.855 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:08 2024-03-25T18:02:19.175 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 998 kB/s | 8.0 MB 00:08 2024-03-25T18:02:25.641 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-03-25T18:02:27.240 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:06 2024-03-25T18:02:37.560 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 748 kB/s | 6.1 MB 00:08 2024-03-25T18:02:39.155 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 754 kB/s | 6.1 MB 00:08 2024-03-25T18:02:39.290 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T18:02:40.492 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T18:02:40.609 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-03-25T18:02:41.614 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T18:02:45.410 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-25T18:02:46.798 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T18:02:47.881 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-25T18:02:47.893 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:02:47.894 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:02:47.895 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:Install 30 Packages 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-03-25T18:02:47.896 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:02:47.897 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-03-25T18:02:47.897 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:02:49.417 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:02:49.447 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:02:49.447 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:02:49.447 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:02:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:02:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:02:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:Install 30 Packages 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 58 M 2024-03-25T18:02:49.451 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:03:08.582 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 273 kB/s | 5.3 MB 00:20 2024-03-25T18:03:08.707 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-25T18:03:10.402 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 264 kB/s | 5.3 MB 00:20 2024-03-25T18:03:10.611 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 121 kB/s | 25 kB 00:00 2024-03-25T18:03:10.998 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 324 kB/s | 742 kB 00:02 2024-03-25T18:03:13.073 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 240 kB/s | 498 kB 00:02 2024-03-25T18:03:13.328 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 273 kB/s | 742 kB 00:02 2024-03-25T18:03:14.662 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 374 kB/s | 498 kB 00:01 2024-03-25T18:03:18.090 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 337 kB/s | 9.7 MB 00:29 2024-03-25T18:03:18.265 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-25T18:03:18.990 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 181 kB/s | 131 kB 00:00 2024-03-25T18:03:19.565 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 282 kB/s | 162 kB 00:00 2024-03-25T18:03:21.449 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 170 kB/s | 320 kB 00:01 2024-03-25T18:03:23.307 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 160 kB/s | 298 kB 00:01 2024-03-25T18:03:23.723 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 238 kB/s | 99 kB 00:00 2024-03-25T18:03:23.849 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 850 kB/s | 106 kB 00:00 2024-03-25T18:03:23.908 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-25T18:03:23.941 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:03:24.000 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:03:24.042 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:03:24.075 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:03:24.140 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:24.278 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:24.465 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:24.485 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:24.530 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:24.619 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:24.661 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:24.751 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:25.233 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:25.560 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:25.731 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:26.086 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:26.106 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:26.167 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:26.229 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:26.251 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:26.412 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:26.766 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:26.904 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 268 kB/s | 9.7 MB 00:37 2024-03-25T18:03:27.033 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 75 kB/s | 221 kB 00:02 2024-03-25T18:03:27.099 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:27.223 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:27.356 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:27.383 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:27.425 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:27.513 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:27.566 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:27.683 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:28.178 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:28.523 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:28.690 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:29.066 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:29.083 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:29.138 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:29.192 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:29.210 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:29.366 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:29.775 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 313 kB/s | 5.1 MB 00:16 2024-03-25T18:03:29.776 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:29.849 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:29.875 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 13 kB/s | 36 kB 00:02 2024-03-25T18:03:29.993 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:30.147 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:30.175 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:30.216 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:30.322 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:30.367 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:30.466 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:30.759 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 350 kB/s | 309 kB 00:00 2024-03-25T18:03:30.947 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:31.272 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:31.359 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-25T18:03:31.451 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-25T18:03:31.451 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:31.542 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-25T18:03:31.676 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 635 kB/s | 67 kB 00:00 2024-03-25T18:03:31.810 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-03-25T18:03:31.837 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:31.854 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:31.918 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-25T18:03:31.928 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:31.982 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:31.999 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:32.069 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T18:03:32.166 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:32.521 INFO:teuthology.orchestra.run.smithi196.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:32.610 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 17 kB/s | 47 kB 00:02 2024-03-25T18:03:34.045 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 6.3 kB/s | 45 kB 00:07 2024-03-25T18:03:34.729 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 192 kB/s | 131 kB 00:00 2024-03-25T18:03:35.388 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 246 kB/s | 162 kB 00:00 2024-03-25T18:03:35.922 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 246 kB/s | 5.1 MB 00:21 2024-03-25T18:03:36.577 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 766 kB/s | 3.4 MB 00:04 2024-03-25T18:03:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 249 kB/s | 320 kB 00:01 2024-03-25T18:03:37.073 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 248 kB/s | 99 kB 00:00 2024-03-25T18:03:37.208 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 783 kB/s | 106 kB 00:00 2024-03-25T18:03:37.242 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 226 kB/s | 298 kB 00:01 2024-03-25T18:03:37.267 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-25T18:03:37.301 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 753 kB/s | 44 kB 00:00 2024-03-25T18:03:37.335 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-25T18:03:37.369 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-25T18:03:37.403 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 615 kB/s | 41 kB 00:00 2024-03-25T18:03:37.437 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:03:37.437 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:03:37.478 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:37.478 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:37.601 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:37.614 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:37.663 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:37.710 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:37.752 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:37.872 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:38.228 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:38.261 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:38.380 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:38.397 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:38.399 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:38.415 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:38.468 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:38.481 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:38.601 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:38.625 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:38.761 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:38.786 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:39.119 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:39.140 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:39.368 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 474 kB/s | 3.1 MB 00:06 2024-03-25T18:03:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:39.632 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:39.935 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:39.960 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:40.001 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:40.027 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:40.133 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:40.151 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:40.190 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:40.222 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:40.223 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:40.279 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:40.411 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 12 kB/s | 36 kB 00:03 2024-03-25T18:03:40.447 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:03:40.501 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:40.528 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 70 kB/s | 221 kB 00:03 2024-03-25T18:03:40.606 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:03:40.647 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:40.745 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:03:40.909 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:41.279 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 427 kB/s | 309 kB 00:00 2024-03-25T18:03:41.316 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:03:41.489 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:41.508 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:03:41.589 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:41.785 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:03:41.896 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-03-25T18:03:41.977 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:42.038 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-25T18:03:42.172 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 363 kB/s | 49 kB 00:00 2024-03-25T18:03:42.281 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 627 kB/s | 67 kB 00:00 2024-03-25T18:03:42.343 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:03:42.432 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-25T18:03:42.575 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-25T18:03:42.751 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-25T18:03:42.858 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:43.212 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:03:43.286 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:43.424 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:03:43.441 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:43.497 INFO:teuthology.orchestra.run.smithi029.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:03:43.585 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 15 kB/s | 47 kB 00:03 2024-03-25T18:03:48.594 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 591 kB/s | 3.4 MB 00:05 2024-03-25T18:03:52.060 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 378 kB/s | 3.1 MB 00:08 2024-03-25T18:04:00.585 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 287 kB/s | 20 MB 01:10 2024-03-25T18:04:00.593 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:00.593 INFO:teuthology.orchestra.run.smithi029.stdout:Total 833 kB/s | 58 MB 01:11 2024-03-25T18:04:00.623 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:04:00.664 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:04:00.664 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:04:01.152 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:04:01.152 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:04:02.623 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:04:02.718 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:04:02.750 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:04:02.776 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:04:02.809 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:04:02.825 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:04:03.028 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:04:03.083 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:04:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:04:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:03.652 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:03.760 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:04:03.787 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:04:03.850 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:04:03.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:04:03.933 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:04:04.003 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:04:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:04:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:04:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:04:04.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:04:04.278 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:04:04.319 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:04:04.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:04:04.390 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:04:04.424 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:04:04.750 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:04:04.781 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:04:04.872 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:04:04.903 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:04:04.953 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:04:05.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:04:05.469 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:04:05.524 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:04:05.604 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:04:05.633 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:04:07.853 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:07.911 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:04:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:04:08.603 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:04:08.603 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:04:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:09.063 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 253 kB/s | 20 MB 01:20 2024-03-25T18:04:09.072 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:09.072 INFO:teuthology.orchestra.run.smithi196.stdout:Total 730 kB/s | 58 MB 01:21 2024-03-25T18:04:09.103 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:04:09.140 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:04:09.140 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:04:09.612 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:04:09.613 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:04:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:04:11.164 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:04:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:04:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:04:11.258 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:04:11.274 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:04:11.497 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:04:11.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:04:11.878 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:11.970 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:12.056 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:04:12.096 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:04:12.159 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:04:12.187 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:04:12.231 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:04:12.299 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:04:12.324 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:04:12.371 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:04:12.405 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:04:12.512 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:04:12.548 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:04:12.588 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:04:12.613 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:04:12.642 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:04:12.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:04:12.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:04:13.021 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:04:13.110 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:04:13.143 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:04:13.186 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:04:13.633 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:04:13.665 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:04:13.728 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:04:13.796 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:04:13.826 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:04:15.946 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:15.989 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:16.547 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:04:16.592 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:04:16.673 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:04:16.674 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:04:16.674 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:04:16.674 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:16.726 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-03-25T18:04:20.029 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-03-25T18:04:20.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:20.973 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:04:20.997 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:04:20.998 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:20.998 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:04:20.998 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:04:20.998 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:04:20.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:21.000 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:04:21.077 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:04:21.077 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:04:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:04:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:04:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:04:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:04:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:04:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:04:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:04:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:04:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:04:23.631 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:23.631 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:23.631 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-03-25T18:04:23.631 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.632 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:04:23.633 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:23.634 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:04:24.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-03-25T18:04:24.629 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:04:24.794 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:04:24.794 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:04:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:04:24.794 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-25T18:04:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:04:24.796 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 48 M 2024-03-25T18:04:24.796 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 204 M 2024-03-25T18:04:24.796 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:04:25.056 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:04:25.156 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T18:04:25.190 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:04:25.424 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 804 kB/s | 187 kB 00:00 2024-03-25T18:04:25.973 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-25T18:04:25.974 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:25.983 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 153 kB/s | 163 kB 00:01 2024-03-25T18:04:26.890 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:04:26.914 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:04:26.914 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:26.914 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:04:26.914 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:04:26.914 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:04:26.915 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:26.916 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:04:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:04:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:04:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:04:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:04:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:04:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:04:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:04:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:04:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:04:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:04:29.225 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.226 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.227 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:29.228 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:04:29.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-03-25T18:04:30.171 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:04:30.325 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:04:30.325 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:30.326 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:04:30.327 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:04:30.327 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-25T18:04:30.327 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 204 M 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:04:30.546 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:04:30.630 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-25T18:04:30.671 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:04:30.755 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-25T18:04:31.197 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 196 kB/s | 163 kB 00:00 2024-03-25T18:06:57.508 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 329 kB/s | 47 MB 02:27 2024-03-25T18:06:57.509 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:57.510 INFO:teuthology.orchestra.run.smithi196.stdout:Total 335 kB/s | 48 MB 02:27 2024-03-25T18:06:57.521 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:06:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:06:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:06:57.914 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:06:57.915 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:06:58.539 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:06:58.607 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:06:58.659 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:06:58.699 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:06:58.747 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:06:58.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:07:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:07:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:07:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:07:05.996 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:07:05.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:07:05.997 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:07:06.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-03-25T18:07:06.893 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:07:07.049 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:07:07.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:07:07.053 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:07:07.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:07:07.055 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-03-25T18:07:07.056 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 M 2024-03-25T18:07:07.056 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:07:07.951 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:07:15.425 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 193 kB/s | 1.5 MB 00:07 2024-03-25T18:07:16.417 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 244 kB/s | 2.1 MB 00:08 2024-03-25T18:07:18.209 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 138 kB/s | 247 kB 00:01 2024-03-25T18:07:18.334 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:07:18.376 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:07:18.409 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-25T18:07:18.460 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-25T18:07:18.494 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:07:18.535 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:07:18.602 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:07:18.636 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:07:18.678 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:07:18.720 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:07:18.920 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 625 kB/s | 125 kB 00:00 2024-03-25T18:07:19.020 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-25T18:07:19.279 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-25T18:07:19.346 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 652 kB/s | 43 kB 00:00 2024-03-25T18:07:19.488 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T18:07:19.613 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T18:07:19.680 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-25T18:07:19.747 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T18:07:19.814 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 348 kB/s | 23 kB 00:00 2024-03-25T18:07:19.881 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-03-25T18:07:19.948 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T18:07:20.018 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-25T18:07:20.082 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T18:07:20.157 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T18:07:20.224 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T18:07:20.308 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-25T18:07:20.375 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-03-25T18:07:20.450 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T18:07:20.525 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:07:20.584 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 344 kB/s | 20 kB 00:00 2024-03-25T18:07:25.725 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 270 kB/s | 4.7 MB 00:17 2024-03-25T18:07:29.378 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 47 MB 03:04 2024-03-25T18:07:29.381 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:29.381 INFO:teuthology.orchestra.run.smithi029.stdout:Total 267 kB/s | 48 MB 03:04 2024-03-25T18:07:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:07:29.413 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:07:29.413 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:07:29.804 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:07:29.805 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:07:30.399 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:07:30.472 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:07:30.519 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:07:30.562 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:07:30.609 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:07:30.636 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:07:35.372 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:07:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:07:39.327 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:07:39.328 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:07:39.328 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:07:39.328 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:07:39.328 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:07:39.328 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:07:39.713 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:07:39.714 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:07:39.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:07:39.714 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:07:40.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-03-25T18:07:40.620 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:07:40.784 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:07:40.786 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:07:40.787 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:07:40.788 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:07:40.789 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2024-03-25T18:07:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:07:40.791 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 30 M 2024-03-25T18:07:40.791 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 107 M 2024-03-25T18:07:40.791 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:07:41.692 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 517 kB/s | 1.5 MB 00:02 2024-03-25T18:07:45.226 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 556 kB/s | 2.1 MB 00:03 2024-03-25T18:07:45.885 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 376 kB/s | 247 kB 00:00 2024-03-25T18:07:46.002 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:07:46.044 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:07:46.078 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-25T18:07:46.112 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:07:46.147 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:07:46.206 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:07:46.282 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:07:46.324 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:07:46.366 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:07:46.408 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:07:46.525 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T18:07:46.576 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 952 kB/s | 151 kB 00:00 2024-03-25T18:07:46.786 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 870 kB/s | 43 kB 00:00 2024-03-25T18:07:46.844 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-25T18:07:46.920 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-25T18:07:46.962 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-25T18:07:47.013 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-25T18:07:47.056 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 535 kB/s | 23 kB 00:00 2024-03-25T18:07:47.099 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-25T18:07:47.141 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 469 kB/s | 20 kB 00:00 2024-03-25T18:07:47.191 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-03-25T18:07:47.242 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T18:07:47.301 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-25T18:07:47.343 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-03-25T18:07:47.394 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-25T18:07:47.444 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-03-25T18:07:47.495 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T18:07:47.562 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-03-25T18:07:47.612 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-25T18:08:02.095 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 235 kB/s | 4.7 MB 00:20 2024-03-25T18:08:14.959 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 289 kB/s | 17 MB 00:59 2024-03-25T18:08:14.963 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:14.963 INFO:teuthology.orchestra.run.smithi196.stdout:Total 447 kB/s | 30 MB 01:07 2024-03-25T18:08:14.985 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:08:15.027 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:08:15.027 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:08:15.427 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:08:15.427 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:08:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:08:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:08:16.869 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:08:16.897 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:08:16.967 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:08:17.004 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:08:17.037 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:08:17.128 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:08:17.215 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:08:17.256 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:08:17.328 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:08:17.382 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:08:17.420 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:08:17.469 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:08:17.525 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:08:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:08:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:08:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:08:17.847 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:08:17.878 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:08:17.953 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:08:17.988 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:08:18.022 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:08:18.062 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:08:18.097 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:08:18.132 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:08:18.220 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:08:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:08:18.335 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:08:18.471 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:08:18.588 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:08:18.746 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:08:18.770 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:08:18.794 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:08:18.794 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:18.795 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:08:18.795 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:08:18.795 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:08:18.795 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:20.279 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:08:20.306 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:08:20.306 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:20.307 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:08:20.307 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:08:20.307 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:08:20.307 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:20.747 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:08:20.770 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:20.973 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:08:20.995 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:08:20.996 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:20.996 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:08:20.996 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:08:20.996 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:08:20.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:21.053 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:08:22.713 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:08:22.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:08:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:08:22.716 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:08:22.717 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:08:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:08:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:08:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:08:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:08:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:23.608 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:23.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-03-25T18:08:24.546 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:24.644 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:24.695 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:24.696 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:08:24.697 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:24.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-03-25T18:08:25.337 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:25.487 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:25.488 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:08:25.489 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:25.489 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 747 k 2024-03-25T18:08:25.489 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 754 k 2024-03-25T18:08:25.489 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:08:29.196 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 202 kB/s | 747 kB 00:03 2024-03-25T18:08:29.196 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:29.197 INFO:teuthology.orchestra.run.smithi196.stdout:Total 202 kB/s | 747 kB 00:03 2024-03-25T18:08:29.197 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:08:29.203 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:08:29.203 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:08:29.210 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:08:29.210 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:08:29.284 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:08:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:29.884 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:30.287 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:30.590 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:30.590 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:30.590 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:08:30.590 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:30.591 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:30.591 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:30.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:08:31.394 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:31.547 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:08:31.548 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:08:31.549 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:31.549 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-03-25T18:08:31.549 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-03-25T18:08:31.549 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:08:32.828 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 114 kB/s | 145 kB 00:01 2024-03-25T18:08:32.828 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:32.828 INFO:teuthology.orchestra.run.smithi196.stdout:Total 114 kB/s | 145 kB 00:01 2024-03-25T18:08:32.829 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:08:32.837 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:08:32.838 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:08:32.893 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:08:32.894 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:08:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:08:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:33.127 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:33.127 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:08:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:34.040 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:34.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-03-25T18:08:34.855 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:34.956 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:35.008 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:35.009 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:08:35.009 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:35.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:08:35.633 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:35.783 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:08:35.784 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-25T18:08:35.785 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:35.786 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.9 M 2024-03-25T18:08:35.786 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-25T18:08:35.786 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:08:36.831 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 22 kB/s | 15 kB 00:00 2024-03-25T18:08:36.956 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 38 kB/s | 31 kB 00:00 2024-03-25T18:08:37.023 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-25T18:08:37.048 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-25T18:08:37.307 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T18:08:37.365 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-03-25T18:08:37.524 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-03-25T18:08:37.583 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-25T18:08:37.633 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 617 kB/s | 31 kB 00:00 2024-03-25T18:08:37.692 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-25T18:08:37.725 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-03-25T18:08:39.303 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 312 kB/s | 17 MB 00:55 2024-03-25T18:08:39.307 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:39.307 INFO:teuthology.orchestra.run.smithi029.stdout:Total 519 kB/s | 30 MB 00:58 2024-03-25T18:08:39.329 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:08:39.376 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:08:39.376 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:08:39.854 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:08:39.854 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:08:41.216 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:08:41.289 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:08:41.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:08:41.385 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:08:41.468 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:08:41.518 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:08:41.666 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:08:41.769 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:08:41.825 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:08:41.909 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:08:41.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:08:41.966 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 313 kB/s | 1.8 MB 00:05 2024-03-25T18:08:41.970 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:41.970 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T18:08:41.991 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:08:42.009 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:08:42.010 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:08:42.010 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:08:42.068 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:08:42.132 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:08:42.194 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:08:42.194 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:08:42.337 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:08:42.394 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:08:42.441 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:08:42.485 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:08:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:08:42.596 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:08:42.635 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:08:42.669 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:08:42.710 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:08:42.712 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:08:42.756 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:08:42.784 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:08:42.791 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:08:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:08:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:08:42.885 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:08:42.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:08:42.931 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:08:42.998 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:08:43.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:08:43.121 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:08:43.176 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:08:43.224 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:08:43.267 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:08:43.296 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:08:43.327 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:08:43.379 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:08:43.572 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:08:43.602 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:08:43.621 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:08:43.652 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:08:43.677 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:44.365 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:08:44.365 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:08:44.365 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:08:44.365 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:08:44.366 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:08:44.729 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:44.730 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:08:44.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:08:45.207 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:08:45.234 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:08:45.234 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:45.234 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:08:45.234 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:08:45.235 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:08:45.235 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:45.538 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:08:45.689 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:08:45.690 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-25T18:08:45.690 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:08:45.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-03-25T18:08:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-03-25T18:08:45.693 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:08:45.700 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:08:45.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:45.939 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 264 kB/s | 32 kB 00:00 2024-03-25T18:08:45.943 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:08:45.966 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:45.998 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 489 kB/s | 17 kB 00:00 2024-03-25T18:08:46.036 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:08:46.040 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 817 kB/s | 34 kB 00:00 2024-03-25T18:08:46.949 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-03-25T18:08:47.240 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-03-25T18:08:47.491 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 871 kB/s | 217 kB 00:00 2024-03-25T18:08:47.674 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.6 MB/s | 4.8 MB 00:00 2024-03-25T18:08:47.983 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-25T18:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-03-25T18:08:48.183 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 985 kB/s | 195 kB 00:00 2024-03-25T18:08:48.403 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:08:48.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:08:48.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:08:48.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:08:48.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:08:48.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:08:48.406 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:08:48.406 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:08:48.442 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 866 kB/s | 245 kB 00:00 2024-03-25T18:08:49.384 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:08:49.385 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:08:49.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:49.387 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:08:49.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-03-25T18:08:50.430 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:08:50.529 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:50.586 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:08:50.587 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:08:50.587 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:08:50.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-03-25T18:08:51.231 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:05 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:08:51.386 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:08:51.387 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:51.388 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:08:51.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:51.388 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 747 k 2024-03-25T18:08:51.388 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 754 k 2024-03-25T18:08:51.388 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:08:55.130 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 200 kB/s | 747 kB 00:03 2024-03-25T18:08:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:Total 200 kB/s | 747 kB 00:03 2024-03-25T18:08:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:08:55.138 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:08:55.138 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:08:55.144 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:08:55.145 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:08:55.228 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:08:55.704 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:55.801 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:56.245 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:56.524 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:08:56.524 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:56.524 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:08:56.524 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:08:56.525 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:56.525 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:08:56.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:08:57.434 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:08:57.589 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:08:57.590 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:57.590 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:08:57.590 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:57.590 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 145 k 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 436 k 2024-03-25T18:08:57.591 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:08:58.672 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 134 kB/s | 145 kB 00:01 2024-03-25T18:08:58.672 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:58.673 INFO:teuthology.orchestra.run.smithi029.stdout:Total 134 kB/s | 145 kB 00:01 2024-03-25T18:08:58.673 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:08:58.683 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:08:58.683 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:08:58.742 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:08:58.742 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:08:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:08:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:59.015 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:59.015 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:08:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:59.976 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:08:59.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:59.977 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:08:59.977 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:59.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:08:59.977 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:00.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-03-25T18:09:00.850 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:00.951 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:01.007 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:01.008 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:09:01.008 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:01.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:09:01.653 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:01.808 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:01.809 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-25T18:09:01.809 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:09:01.809 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:09:01.810 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:01.811 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.9 M 2024-03-25T18:09:01.812 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-03-25T18:09:01.812 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:09:02.583 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T18:09:02.608 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-25T18:09:02.759 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T18:09:02.801 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T18:09:03.168 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T18:09:03.244 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 261 kB/s | 19 kB 00:00 2024-03-25T18:09:03.595 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.8 MB/s | 2.0 MB 00:00 2024-03-25T18:09:03.670 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-25T18:09:03.729 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-25T18:09:03.806 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-25T18:09:04.740 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 436 kB/s | 863 kB 00:01 2024-03-25T18:09:08.615 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 276 kB/s | 1.8 MB 00:06 2024-03-25T18:09:08.621 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:08.621 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T18:09:08.641 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:09:08.662 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:09:08.662 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:09:08.855 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:09:08.855 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:09:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:09:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:09:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:09:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:09:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:09:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:09:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:09:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:09:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:09:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:09:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:09:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:09:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:09:11.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:09:11.544 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:09:11.544 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:11.887 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:11.888 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:12.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:09:12.765 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:12.925 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:09:12.926 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-03-25T18:09:12.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:12.928 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-03-25T18:09:12.928 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-03-25T18:09:12.928 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:09:13.815 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-25T18:09:13.849 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 511 kB/s | 17 kB 00:00 2024-03-25T18:09:13.883 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:09:14.075 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T18:09:14.244 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T18:09:14.320 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T18:09:14.345 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-25T18:09:14.796 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-25T18:09:15.063 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-25T18:09:15.255 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 426 kB/s | 195 kB 00:00 2024-03-25T18:09:15.356 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 841 kB/s | 245 kB 00:00 2024-03-25T18:09:17.138 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 242 kB/s | 7.4 MB 00:31 2024-03-25T18:09:17.140 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:17.140 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.2 MB/s | 37 MB 00:31 2024-03-25T18:09:17.181 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:17.207 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:17.207 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:17.560 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:17.561 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:18.343 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:09:18.375 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:09:18.478 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:09:18.520 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:09:18.696 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:09:18.727 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:09:18.808 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:09:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:09:19.581 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:09:20.818 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:09:21.835 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:09:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:09:24.387 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:09:24.388 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:09:24.832 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:09:24.832 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:24.832 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:24.833 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:25.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:09:25.776 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:40 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:25.926 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:09:25.927 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:25.928 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:25.929 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-03-25T18:09:25.929 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:25.929 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-25T18:09:25.929 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-25T18:09:25.929 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:26.086 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 216 kB/s | 24 kB 00:00 2024-03-25T18:09:26.136 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-25T18:09:26.220 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-25T18:09:26.262 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-25T18:09:26.329 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.9 MB/s | 211 kB 00:00 2024-03-25T18:09:26.354 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 465 kB/s | 43 kB 00:00 2024-03-25T18:09:26.613 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 114 kB/s | 32 kB 00:00 2024-03-25T18:09:26.638 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-25T18:09:26.772 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T18:09:26.847 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-25T18:09:26.939 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 978 kB/s | 90 kB 00:00 2024-03-25T18:09:27.106 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 46 kB/s | 52 kB 00:01 2024-03-25T18:09:27.139 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-25T18:09:27.143 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:27.143 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-25T18:09:27.143 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:27.156 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:27.156 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:27.246 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:27.246 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:27.764 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:27.927 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:09:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:09:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:09:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:09:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:09:28.204 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:09:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:09:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:09:29.073 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:09:29.125 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:09:29.159 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:09:29.207 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:09:30.001 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:09:30.001 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:09:30.001 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:09:30.001 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:09:30.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:09:30.390 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:09:30.390 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:30.390 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:30.390 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:09:30.391 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:30.392 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:30.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:09:31.210 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:46 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:31.362 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:31.362 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:31.363 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:09:31.364 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-25T18:09:31.364 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:31.364 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-03-25T18:09:31.364 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-03-25T18:09:31.364 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:31.835 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T18:09:31.869 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 777 kB/s | 248 kB 00:00 2024-03-25T18:09:32.102 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 275 kB/s | 58 kB 00:00 2024-03-25T18:09:32.161 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 873 kB/s | 51 kB 00:00 2024-03-25T18:09:32.311 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:09:32.403 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 161 kB/s | 138 kB 00:00 2024-03-25T18:09:32.405 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:32.406 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.5 MB/s | 6.8 MB 00:01 2024-03-25T18:09:32.418 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:32.428 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:32.428 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:32.517 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:32.517 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:32.646 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 399 kB/s | 7.4 MB 00:18 2024-03-25T18:09:32.650 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:32.650 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.9 MB/s | 37 MB 00:19 2024-03-25T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:09:32.822 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:32.987 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:09:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:09:33.103 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:09:33.103 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:09:33.553 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:09:33.640 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:09:33.702 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:09:33.726 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:09:33.828 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:09:33.904 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:09:33.944 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:09:34.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:09:34.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:09:34.294 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:09:34.324 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:09:34.423 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:09:34.531 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:34.837 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:35.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-03-25T18:09:35.114 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:09:35.240 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:09:35.657 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:50 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:35.809 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:35.810 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-25T18:09:35.811 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:35.811 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 964 k 2024-03-25T18:09:35.811 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-03-25T18:09:35.811 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:35.938 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 618 kB/s | 79 kB 00:00 2024-03-25T18:09:36.546 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:09:37.640 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:09:37.672 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:09:37.680 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 474 kB/s | 885 kB 00:01 2024-03-25T18:09:37.682 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:37.682 INFO:teuthology.orchestra.run.smithi196.stdout:Total 515 kB/s | 964 kB 00:01 2024-03-25T18:09:37.682 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:37.706 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:37.706 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:37.782 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:37.783 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:37.971 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:09:38.172 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:38.817 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:38.817 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:39.104 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:39.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-03-25T18:09:39.969 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:40.122 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:40.123 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 332 k 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.5 M 2024-03-25T18:09:40.124 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:40.235 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 625 kB/s | 69 kB 00:00 2024-03-25T18:09:41.627 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 176 kB/s | 264 kB 00:01 2024-03-25T18:09:41.628 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:41.629 INFO:teuthology.orchestra.run.smithi196.stdout:Total 221 kB/s | 332 kB 00:01 2024-03-25T18:09:41.629 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:41.639 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:41.639 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:41.682 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:41.682 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:41.833 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:09:41.924 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:09:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:09:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:09:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:09:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:09:42.012 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:09:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:09:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:09:42.069 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:09:42.069 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:09:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:42.405 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:09:42.405 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:42.405 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:09:42.405 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:42.405 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:42.406 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:42.804 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:09:42.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:09:43.176 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:43.177 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:43.396 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:57 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:43.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-03-25T18:09:43.552 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:43.553 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:43.553 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:09:43.553 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:43.553 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:09:43.553 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:09:43.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-03-25T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:09:43.729 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 237 kB/s | 24 kB 00:00 2024-03-25T18:09:43.790 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-25T18:09:43.840 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-25T18:09:43.866 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:09:43.908 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-25T18:09:43.941 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 574 kB/s | 43 kB 00:00 2024-03-25T18:09:44.085 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:06:59 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:44.134 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-25T18:09:44.201 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-25T18:09:44.233 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:09:44.234 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:44.235 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-03-25T18:09:44.235 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-03-25T18:09:44.235 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:44.327 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.0 MB/s | 203 kB 00:00 2024-03-25T18:09:44.369 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 70 kB/s | 52 kB 00:00 2024-03-25T18:09:44.461 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T18:09:44.487 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 371 kB/s | 59 kB 00:00 2024-03-25T18:09:44.512 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 631 kB/s | 90 kB 00:00 2024-03-25T18:09:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T18:09:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:09:44.531 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:09:44.531 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:09:44.631 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:09:44.631 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:09:44.916 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 186 kB/s | 127 kB 00:00 2024-03-25T18:09:44.916 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:44.917 INFO:teuthology.orchestra.run.smithi196.stdout:Total 186 kB/s | 127 kB 00:00 2024-03-25T18:09:44.917 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:44.923 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:44.923 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:44.982 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:44.982 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:45.132 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:45.159 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:09:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:45.337 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:09:45.394 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:09:45.458 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:09:45.494 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:09:45.545 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:09:45.644 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:09:45.684 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:45.761 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:09:45.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:45.959 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:46.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-03-25T18:09:46.590 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:09:46.652 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:09:46.686 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:09:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:09:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:09:46.782 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:01 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:46.880 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:46.937 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:46.938 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:46.938 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:47.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-03-25T18:09:47.571 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:02 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:47.720 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:09:47.721 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:47.722 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-03-25T18:09:47.722 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-03-25T18:09:47.722 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:09:47.899 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:09:47.900 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:09:47.900 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:09:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:09:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:09:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:09:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:48.256 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:48.261 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T18:09:48.261 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:48.262 INFO:teuthology.orchestra.run.smithi196.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:09:48.262 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:48.266 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:48.266 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:48.308 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:48.308 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:48.433 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:48.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:09:48.559 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:48.903 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:49.132 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:49.167 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:49.289 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:09:49.290 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:49.291 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2024-03-25T18:09:49.292 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-03-25T18:09:49.292 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:09:49.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-03-25T18:09:49.823 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T18:09:49.892 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:04 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:49.989 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:50.040 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:50.041 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:50.041 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:50.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-03-25T18:09:50.123 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T18:09:50.182 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 985 kB/s | 58 kB 00:00 2024-03-25T18:09:50.266 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 609 kB/s | 51 kB 00:00 2024-03-25T18:09:50.408 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 153 kB/s | 138 kB 00:00 2024-03-25T18:09:50.525 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-03-25T18:09:50.527 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:50.527 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-25T18:09:50.542 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:09:50.554 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:09:50.554 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:09:50.651 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:09:50.652 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:09:50.664 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:05 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:50.764 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:50.815 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:50.816 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:50.816 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:50.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-03-25T18:09:50.956 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:09:51.140 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:09:51.204 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:09:51.429 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:06 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:51.526 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:51.577 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:51.578 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:51.578 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:51.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-03-25T18:09:51.768 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:09:51.862 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:09:51.932 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:09:51.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:09:52.184 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:52.280 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:52.332 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:52.333 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:52.333 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:52.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-03-25T18:09:52.957 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:53.063 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:09:53.099 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:09:53.118 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:53.119 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:53.119 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:53.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:53.398 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:53.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-03-25T18:09:53.753 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:08 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:53.853 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:09:53.907 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:53.909 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T18:09:53.909 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:53.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-03-25T18:09:54.279 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:08 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:09:54.434 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-25T18:09:54.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:54.436 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 964 k 2024-03-25T18:09:54.436 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.8 M 2024-03-25T18:09:54.436 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:09:54.524 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:09 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:54.547 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 708 kB/s | 79 kB 00:00 2024-03-25T18:09:54.672 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:54.673 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:09:54.674 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:54.674 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-03-25T18:09:54.674 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-03-25T18:09:54.674 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:55.489 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 105 kB/s | 85 kB 00:00 2024-03-25T18:09:55.489 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:55.490 INFO:teuthology.orchestra.run.smithi196.stdout:Total 104 kB/s | 85 kB 00:00 2024-03-25T18:09:55.490 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:09:55.498 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:09:55.498 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:09:55.554 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:09:55.555 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:09:55.694 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:09:55.795 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:56.238 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:56.574 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:56.574 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:56.574 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:09:56.574 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:56.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:56.575 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:09:56.793 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-03-25T18:09:57.363 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:12 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:09:57.519 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:09:57.520 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:09:57.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:09:57.521 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-03-25T18:09:57.521 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-03-25T18:09:57.521 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:09:57.706 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 271 kB/s | 885 kB 00:03 2024-03-25T18:09:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:Total 294 kB/s | 964 kB 00:03 2024-03-25T18:09:57.708 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:09:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:09:57.733 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:09:57.813 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:09:57.814 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:09:57.993 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:09:58.139 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:09:58.212 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:09:59.285 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:09:59.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2024-03-25T18:10:00.193 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:14 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:00.351 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:00.351 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:00.351 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 332 k 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.5 M 2024-03-25T18:10:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:00.469 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 588 kB/s | 69 kB 00:00 2024-03-25T18:10:01.563 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 218 kB/s | 264 kB 00:01 2024-03-25T18:10:01.565 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:01.565 INFO:teuthology.orchestra.run.smithi029.stdout:Total 274 kB/s | 332 kB 00:01 2024-03-25T18:10:01.565 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:01.575 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:01.576 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:01.618 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:01.619 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:01.964 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:10:02.025 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:10:02.026 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:10:02.026 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:10:02.026 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:02.839 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:03.167 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:03.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-03-25T18:10:04.137 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:18 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:04.292 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 127 k 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 456 k 2024-03-25T18:10:04.293 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:05.042 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 420 kB/s | 3.1 MB 00:07 2024-03-25T18:10:05.042 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:05.043 INFO:teuthology.orchestra.run.smithi196.stdout:Total 420 kB/s | 3.1 MB 00:07 2024-03-25T18:10:05.043 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:10:05.053 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:10:05.054 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:10:05.130 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:10:05.130 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:10:05.287 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 127 kB/s | 127 kB 00:00 2024-03-25T18:10:05.288 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:05.288 INFO:teuthology.orchestra.run.smithi029.stdout:Total 127 kB/s | 127 kB 00:00 2024-03-25T18:10:05.288 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:05.295 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:05.295 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:05.355 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:05.355 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:05.501 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:05.568 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:10:05.614 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:05.634 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:05.658 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:05.658 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:10:05.658 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:10:05.658 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:10:05.658 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:10:05.659 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:06.150 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:06.434 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:06.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:06.763 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:10:07.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-03-25T18:10:07.278 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:21 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:07.379 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:07.435 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:07.436 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:07.437 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:07.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-03-25T18:10:07.596 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:22 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:10:07.745 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:07.746 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-03-25T18:10:07.747 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:10:08.087 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:22 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:08.246 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:08.247 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 k 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 140 k 2024-03-25T18:10:08.248 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:08.882 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 49 kB/s | 31 kB 00:00 2024-03-25T18:10:08.883 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:08.883 INFO:teuthology.orchestra.run.smithi029.stdout:Total 49 kB/s | 31 kB 00:00 2024-03-25T18:10:08.883 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:08.911 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 147 kB/s | 171 kB 00:01 2024-03-25T18:10:08.911 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:08.912 INFO:teuthology.orchestra.run.smithi196.stdout:Total 147 kB/s | 171 kB 00:01 2024-03-25T18:10:08.912 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:10:08.916 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:08.916 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:08.921 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:10:08.921 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:10:08.981 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:10:08.981 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:10:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:09.137 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:10:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.650 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:09.864 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:09.938 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:09.938 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:09.938 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:10:09.938 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:09.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:09.939 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:10:10.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-03-25T18:10:10.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:10:10.610 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:24 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:10.710 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:10.755 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:07:25 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:10:10.765 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:10.766 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:10.766 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:10.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-03-25T18:10:10.911 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-25T18:10:10.912 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:10:10.913 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 91 M 2024-03-25T18:10:10.913 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 498 M 2024-03-25T18:10:10.913 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:10:11.386 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:25 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:11.486 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:11.540 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:11.541 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:11.542 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:11.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-03-25T18:10:12.162 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:26 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:12.263 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:12.318 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:12.319 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:12.319 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:12.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-03-25T18:10:12.943 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:26 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:13.043 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:13.097 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:13.098 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:13.098 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:13.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-03-25T18:10:13.724 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:27 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:13.823 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:13.878 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:13.879 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:13.879 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:13.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-03-25T18:10:14.500 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:28 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:14.600 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:10:14.655 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:14.656 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:10:14.656 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:14.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-03-25T18:10:15.273 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:29 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:15.429 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:15.429 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:10:15.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:15.431 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 85 k 2024-03-25T18:10:15.431 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 231 k 2024-03-25T18:10:15.431 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:16.391 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 89 kB/s | 85 kB 00:00 2024-03-25T18:10:16.392 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:16.392 INFO:teuthology.orchestra.run.smithi029.stdout:Total 89 kB/s | 85 kB 00:00 2024-03-25T18:10:16.392 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:16.401 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:16.401 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:16.462 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:16.462 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:16.604 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:16.726 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:17.181 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:17.491 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:17.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-03-25T18:10:18.305 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:32 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:18.464 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:18.465 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:10:18.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:18.466 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.1 M 2024-03-25T18:10:18.466 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 13 M 2024-03-25T18:10:18.466 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 382 kB/s | 3.1 MB 00:08 2024-03-25T18:10:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:Total 382 kB/s | 3.1 MB 00:08 2024-03-25T18:10:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:27.292 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:27.369 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:27.393 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:27.393 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:10:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:10:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:10:27.394 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:10:27.394 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:28.617 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:28.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-03-25T18:10:29.472 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:43 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:29.627 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:10:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 171 k 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 494 k 2024-03-25T18:10:29.629 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:10:30.619 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 173 kB/s | 171 kB 00:00 2024-03-25T18:10:30.619 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:30.620 INFO:teuthology.orchestra.run.smithi029.stdout:Total 173 kB/s | 171 kB 00:00 2024-03-25T18:10:30.620 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:10:30.629 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:10:30.629 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:10:30.692 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:10:30.692 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:10:30.851 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:10:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:31.439 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:31.735 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:31.735 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:31.736 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:10:31.736 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:31.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:31.736 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:10:31.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:10:32.566 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:46 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:10:32.723 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:10:32.723 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-25T18:10:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:10:32.725 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 91 M 2024-03-25T18:10:32.725 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 498 M 2024-03-25T18:10:32.725 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:11:06.897 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 411 kB/s | 22 MB 00:55 2024-03-25T18:11:16.239 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 528 kB/s | 22 MB 00:43 2024-03-25T18:14:09.568 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 292 kB/s | 68 MB 03:58 2024-03-25T18:14:09.569 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:14:09.569 INFO:teuthology.orchestra.run.smithi196.stdout:Total 389 kB/s | 91 MB 03:58 2024-03-25T18:14:09.569 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:14:09.573 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:14:09.574 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:14:10.164 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:14:10.165 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:14:13.261 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:14:20.423 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:14:20.501 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:14:23.822 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:14:23.823 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:14:24.098 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:14:24.098 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:14:24.098 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:14:24.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:24.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:24.099 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:14:24.099 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:14:24.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:14:24.715 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 301 kB/s | 68 MB 03:51 2024-03-25T18:14:24.716 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:14:24.717 INFO:teuthology.orchestra.run.smithi029.stdout:Total 400 kB/s | 91 MB 03:51 2024-03-25T18:14:24.717 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:14:24.721 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:14:24.721 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:14:24.886 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:11:39 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:14:25.036 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 416 M 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.7 G 2024-03-25T18:14:25.037 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:14:25.330 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:14:25.331 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:14:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:14:36.113 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:14:36.209 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:14:43.002 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:14:43.003 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:14:43.274 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:14:43.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:14:44.091 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:11:58 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:14:44.248 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:14:44.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:14:44.250 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 416 M 2024-03-25T18:14:44.250 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.7 G 2024-03-25T18:14:44.250 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:28:22.412 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 521 kB/s | 416 MB 13:38 2024-03-25T18:28:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:28:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:Total 521 kB/s | 416 MB 13:38 2024-03-25T18:28:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:28:22.417 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:28:22.417 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:28:25.003 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:28:25.003 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:29:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:29:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:29:20.739 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:29:20.994 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:29:20.994 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:20.994 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:29:20.995 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:20.995 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:20.995 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:29:21.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:29:21.876 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:26:35 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:29:22.031 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:29:22.032 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:29:22.032 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:29:22.032 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.4 M 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 6.4 M 2024-03-25T18:29:22.033 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:29:26.278 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 344 kB/s | 1.4 MB 00:04 2024-03-25T18:29:26.279 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:29:26.279 INFO:teuthology.orchestra.run.smithi029.stdout:Total 344 kB/s | 1.4 MB 00:04 2024-03-25T18:29:26.279 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:29:26.283 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:29:26.284 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:29:26.330 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:29:26.331 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:29:26.604 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:29:26.713 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:29:27.184 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:29:27.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:29:27.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:27.461 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:29:27.461 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:27.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:27.462 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:29:27.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:29:28.230 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:26:42 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:29:28.385 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:29:28.386 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:29:28.387 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:29:28.387 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 176 M 2024-03-25T18:29:28.387 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 753 M 2024-03-25T18:29:28.387 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:31:12.487 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 423 kB/s | 416 MB 16:47 2024-03-25T18:31:12.487 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:12.487 INFO:teuthology.orchestra.run.smithi196.stdout:Total 423 kB/s | 416 MB 16:47 2024-03-25T18:31:12.487 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:31:12.490 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:31:12.490 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:31:14.951 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:31:14.951 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:31:56.973 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:31:57.078 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:32:03.951 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:32:04.224 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:32:04.224 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:04.224 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:32:04.224 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:04.225 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:04.225 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:32:04.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:32:05.060 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:29:20 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:32:05.210 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-25T18:32:05.211 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:32:05.212 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:05.212 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 1.4 M 2024-03-25T18:32:05.212 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 6.4 M 2024-03-25T18:32:05.212 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:32:08.040 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 517 kB/s | 1.4 MB 00:02 2024-03-25T18:32:08.040 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:08.041 INFO:teuthology.orchestra.run.smithi196.stdout:Total 516 kB/s | 1.4 MB 00:02 2024-03-25T18:32:08.041 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:32:08.045 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:32:08.045 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:32:08.089 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:32:08.090 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:32:08.342 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:32:08.438 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:32:08.848 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:32:09.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:32:09.109 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:09.110 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:32:09.110 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:09.110 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:09.110 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:32:09.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:32:09.863 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:29:24 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:32:10.013 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:32:10.014 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:32:10.015 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:32:10.015 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:32:10.015 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 176 M 2024-03-25T18:32:10.015 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 753 M 2024-03-25T18:32:10.015 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:36:20.814 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 436 kB/s | 176 MB 06:52 2024-03-25T18:36:20.815 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:20.815 INFO:teuthology.orchestra.run.smithi029.stdout:Total 436 kB/s | 176 MB 06:52 2024-03-25T18:36:20.815 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:36:20.817 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:36:20.817 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:36:21.927 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:36:21.927 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:36:41.043 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:36:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:36:51.176 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:36:51.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:36:51.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:36:51.451 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:36:51.451 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:51.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:36:51.451 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:36:51.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:36:52.264 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:34:06 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:36:52.419 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:36:52.420 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:36:52.421 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:36:52.421 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:36:52.421 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 672 M 2024-03-25T18:36:52.421 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.1 G 2024-03-25T18:36:52.421 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T18:40:37.568 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 354 kB/s | 176 MB 08:27 2024-03-25T18:40:37.568 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:40:37.568 INFO:teuthology.orchestra.run.smithi196.stdout:Total 354 kB/s | 176 MB 08:27 2024-03-25T18:40:37.568 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T18:40:37.571 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T18:40:37.571 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T18:40:38.629 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T18:40:38.629 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T18:40:56.970 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T18:40:57.071 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:41:01.525 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:41:01.787 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:41:01.787 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:41:01.788 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T18:41:01.788 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:41:01.788 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:41:01.788 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T18:41:02.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:41:02.585 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:38:17 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T18:41:02.734 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T18:41:02.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T18:41:02.736 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 672 M 2024-03-25T18:41:02.736 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 3.1 G 2024-03-25T18:41:02.736 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T18:58:02.925 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 542 kB/s | 672 MB 21:10 2024-03-25T18:58:02.925 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:02.925 INFO:teuthology.orchestra.run.smithi029.stdout:Total 542 kB/s | 672 MB 21:10 2024-03-25T18:58:02.925 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T18:58:02.930 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T18:58:02.930 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T18:58:07.086 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T18:58:07.086 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T18:59:22.587 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T18:59:23.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:41.180 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:41.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:41.441 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:59:41.441 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T18:59:41.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:41.441 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:59:41.441 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:59:41.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:59:42.337 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:56:56 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:59:42.437 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:59:42.495 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:59:42.496 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-25T18:59:42.496 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T18:59:42.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:59:43.144 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:56:57 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T18:59:43.299 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T18:59:43.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T18:59:43.301 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 20 M 2024-03-25T18:59:43.301 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 103 M 2024-03-25T18:59:43.301 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:00:29.859 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 433 kB/s | 20 MB 00:46 2024-03-25T19:00:29.860 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:29.860 INFO:teuthology.orchestra.run.smithi029.stdout:Total 433 kB/s | 20 MB 00:46 2024-03-25T19:00:29.860 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:00:29.864 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:00:29.865 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:00:30.023 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:00:30.023 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:00:32.375 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:00:32.487 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:34.268 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:00:34.544 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:00:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:00:34.544 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:00:34.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:00:35.321 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:57:49 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:00:35.477 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 48 M 2024-03-25T19:00:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 173 M 2024-03-25T19:00:35.479 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:02:35.050 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 409 kB/s | 48 MB 01:59 2024-03-25T19:02:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:Total 409 kB/s | 48 MB 01:59 2024-03-25T19:02:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:02:35.054 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:02:35.054 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:02:35.381 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:02:35.381 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:02:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:02:40.503 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:43.046 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:43.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:02:43.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:02:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:43.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:02:43.318 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:02:43.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:02:44.088 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:59:58 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:02:44.243 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:02:44.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:02:44.245 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 99 M 2024-03-25T19:02:44.245 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 449 M 2024-03-25T19:02:44.245 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:07:18.096 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 99 MB 04:33 2024-03-25T19:07:18.096 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:18.096 INFO:teuthology.orchestra.run.smithi029.stdout:Total 368 kB/s | 99 MB 04:33 2024-03-25T19:07:18.096 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:07:18.099 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:07:18.099 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:07:18.737 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:07:18.737 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:07:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:07:29.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:35.772 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:36.040 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:07:36.041 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:07:36.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:07:36.825 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:04:50 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:07:36.981 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:07:36.982 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 287 M 2024-03-25T19:07:36.983 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.2 G 2024-03-25T19:07:36.983 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:14:12.066 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 346 kB/s | 672 MB 33:09 2024-03-25T19:14:12.066 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:14:12.067 INFO:teuthology.orchestra.run.smithi196.stdout:Total 346 kB/s | 672 MB 33:09 2024-03-25T19:14:12.067 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:14:12.071 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:14:12.071 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:14:16.037 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:14:16.037 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:15:26.795 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:15:27.306 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:36.011 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:36.274 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:36.274 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:15:36.274 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:15:36.275 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:36.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:15:36.275 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:15:36.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:15:37.146 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:12:52 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:15:37.243 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:15:37.294 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:15:37.296 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-25T19:15:37.296 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:15:37.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:15:37.930 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:12:52 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 20 M 2024-03-25T19:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 103 M 2024-03-25T19:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:16:32.465 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 371 kB/s | 20 MB 00:54 2024-03-25T19:16:32.466 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:32.466 INFO:teuthology.orchestra.run.smithi196.stdout:Total 371 kB/s | 20 MB 00:54 2024-03-25T19:16:32.467 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:16:32.470 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:16:32.471 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:16:32.618 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:16:32.619 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:16:34.835 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:16:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:16:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:16:36.104 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:16:36.104 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:16:36.104 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:16:36.105 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:36.105 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:16:36.105 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:16:36.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:16:36.846 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:13:51 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:16:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 173 M 2024-03-25T19:16:36.995 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:17:31.099 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 494 kB/s | 287 MB 09:54 2024-03-25T19:17:31.099 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:31.099 INFO:teuthology.orchestra.run.smithi029.stdout:Total 494 kB/s | 287 MB 09:54 2024-03-25T19:17:31.100 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:17:31.103 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:17:31.103 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:17:32.894 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:17:32.894 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:18:03.981 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:18:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:20.558 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:20.849 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:18:21.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:18:21.660 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:15:35 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:18:21.814 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:18:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:21.816 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 5.9 M 2024-03-25T19:18:21.816 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 28 M 2024-03-25T19:18:21.816 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:18:35.128 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 457 kB/s | 5.9 MB 00:13 2024-03-25T19:18:35.129 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:35.130 INFO:teuthology.orchestra.run.smithi029.stdout:Total 457 kB/s | 5.9 MB 00:13 2024-03-25T19:18:35.130 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:18:35.134 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:18:35.134 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:18:35.210 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:18:35.210 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:18:36.089 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:36.918 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:37.204 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:37.204 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:37.204 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:18:37.204 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:37.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:37.205 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:18:37.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:18:37.556 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 405 kB/s | 48 MB 02:00 2024-03-25T19:18:37.557 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:37.557 INFO:teuthology.orchestra.run.smithi196.stdout:Total 405 kB/s | 48 MB 02:00 2024-03-25T19:18:37.557 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:18:37.561 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:18:37.561 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:18:37.874 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:18:37.874 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:18:37.990 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:15:51 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:18:38.148 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:38.149 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:18:38.150 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:38.150 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 109 k 2024-03-25T19:18:38.150 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 440 k 2024-03-25T19:18:38.150 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:18:39.128 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 111 kB/s | 109 kB 00:00 2024-03-25T19:18:39.128 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:39.129 INFO:teuthology.orchestra.run.smithi029.stdout:Total 111 kB/s | 109 kB 00:00 2024-03-25T19:18:39.129 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:18:39.133 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:18:39.133 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:18:39.173 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:18:39.173 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:18:39.291 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:18:39.396 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:39.850 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:40.137 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:18:40.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:18:40.919 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:15:54 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:18:41.074 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 5.4 M 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 25 M 2024-03-25T19:18:41.075 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:18:42.608 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:18:42.696 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:44.341 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:44.341 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:18:44.341 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:18:44.342 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:44.342 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:18:44.342 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:18:44.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:18:45.082 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:16:00 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:18:45.231 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:18:45.231 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:18:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:18:45.231 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:18:45.231 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 99 M 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 449 M 2024-03-25T19:18:45.232 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:18:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 521 kB/s | 5.4 MB 00:10 2024-03-25T19:18:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:51.595 INFO:teuthology.orchestra.run.smithi029.stdout:Total 521 kB/s | 5.4 MB 00:10 2024-03-25T19:18:51.595 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:18:51.599 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:18:51.599 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:18:51.671 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:18:51.672 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:18:52.370 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:18:52.479 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:53.232 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:53.519 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:18:53.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2-debuginfo 2024-03-25T19:18:54.332 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:16:08 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:18:54.486 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:18:54.487 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 44 M 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 253 M 2024-03-25T19:18:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:20:32.470 INFO:teuthology.orchestra.run.smithi029.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 463 kB/s | 44 MB 01:37 2024-03-25T19:20:32.471 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:32.471 INFO:teuthology.orchestra.run.smithi029.stdout:Total 463 kB/s | 44 MB 01:37 2024-03-25T19:20:32.471 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:20:32.474 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:20:32.475 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:20:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:20:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:20:38.156 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:20:38.265 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:41.919 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:42.198 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:42.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:20:42.199 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:20:42.199 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:42.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:20:42.199 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:20:42.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:20:42.987 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:17:56 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:20:43.144 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:20:43.145 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:20:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:20:43.146 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:20:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 52 M 2024-03-25T19:20:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 308 M 2024-03-25T19:20:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:21:45.653 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 559 kB/s | 99 MB 03:00 2024-03-25T19:21:45.654 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:45.654 INFO:teuthology.orchestra.run.smithi196.stdout:Total 559 kB/s | 99 MB 03:00 2024-03-25T19:21:45.654 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:21:45.658 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:21:45.658 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:21:46.265 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:21:46.265 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:21:56.534 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:21:56.641 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:59.463 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:59.726 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:59.726 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:21:59.727 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:21:59.727 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:59.727 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:21:59.727 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:21:59.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:22:00.475 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:19:15 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:22:00.623 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:22:00.624 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:22:00.625 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:22:00.625 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 287 M 2024-03-25T19:22:00.625 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.2 G 2024-03-25T19:22:00.625 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:22:25.130 INFO:teuthology.orchestra.run.smithi029.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 521 kB/s | 52 MB 01:41 2024-03-25T19:22:25.131 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:25.131 INFO:teuthology.orchestra.run.smithi029.stdout:Total 521 kB/s | 52 MB 01:41 2024-03-25T19:22:25.131 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:25.134 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:25.134 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:25.496 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:25.496 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:31.827 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:31.942 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:36.535 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:36.535 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:36.535 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:22:36.535 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:36.536 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:36.536 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:22:36.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:22:37.311 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:19:51 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:22:37.468 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:22:37.468 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:37.468 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:22:37.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:37.470 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 615 k 2024-03-25T19:22:37.470 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.1 M 2024-03-25T19:22:37.470 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:22:39.496 INFO:teuthology.orchestra.run.smithi029.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 303 kB/s | 615 kB 00:02 2024-03-25T19:22:39.497 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:39.497 INFO:teuthology.orchestra.run.smithi029.stdout:Total 303 kB/s | 615 kB 00:02 2024-03-25T19:22:39.497 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:39.501 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:39.502 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:39.544 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:39.545 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:39.718 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:39.841 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:40.302 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:40.582 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:22:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:40.582 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:22:40.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:22:41.398 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:19:55 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:22:41.556 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:22:41.556 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 771 k 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.7 M 2024-03-25T19:22:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:22:43.588 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 380 kB/s | 771 kB 00:02 2024-03-25T19:22:43.589 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:43.589 INFO:teuthology.orchestra.run.smithi029.stdout:Total 379 kB/s | 771 kB 00:02 2024-03-25T19:22:43.589 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:43.594 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:43.594 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:43.637 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:43.637 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:43.950 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:44.412 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:44.699 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:44.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:44.700 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:22:44.700 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:44.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:44.700 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:22:44.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:22:45.489 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:19:59 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:22:45.645 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:22:45.645 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:45.645 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:22:45.645 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:22:45.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:45.647 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.1 M 2024-03-25T19:22:45.647 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 4.3 M 2024-03-25T19:22:45.647 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:22:51.446 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 198 kB/s | 1.1 MB 00:05 2024-03-25T19:22:51.447 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:51.447 INFO:teuthology.orchestra.run.smithi029.stdout:Total 198 kB/s | 1.1 MB 00:05 2024-03-25T19:22:51.448 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:51.452 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:51.452 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:51.497 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:51.498 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:51.724 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:51.832 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:52.596 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:52.597 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:22:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:52.597 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:52.597 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:22:52.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:22:53.387 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:20:07 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:22:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:22:53.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 337 k 2024-03-25T19:22:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.0 M 2024-03-25T19:22:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:22:55.180 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 206 kB/s | 337 kB 00:01 2024-03-25T19:22:55.180 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:55.180 INFO:teuthology.orchestra.run.smithi029.stdout:Total 206 kB/s | 337 kB 00:01 2024-03-25T19:22:55.180 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:55.184 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:55.185 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:55.226 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:55.226 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:55.474 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:55.892 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:56.215 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:22:56.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:22:57.002 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:20:11 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:22:57.157 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 509 k 2024-03-25T19:22:57.158 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.5 M 2024-03-25T19:22:57.159 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:22:59.135 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 257 kB/s | 509 kB 00:01 2024-03-25T19:22:59.136 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:59.136 INFO:teuthology.orchestra.run.smithi029.stdout:Total 257 kB/s | 509 kB 00:01 2024-03-25T19:22:59.136 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:22:59.141 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:22:59.141 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:22:59.183 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:22:59.184 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:22:59.354 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:22:59.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:59.909 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:00.197 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:00.197 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:23:00.198 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:23:00.198 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:00.198 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:23:00.198 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:23:00.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:23:00.964 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:20:14 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:23:01.122 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 52 M 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 298 M 2024-03-25T19:23:01.123 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 618 kB/s | 52 MB 01:25 2024-03-25T19:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:Total 618 kB/s | 52 MB 01:25 2024-03-25T19:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:24:26.910 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:24:26.910 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:24:33.193 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:24:33.315 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:37.551 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:37.826 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:24:38.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:24:38.601 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 1:21:52 ago on Mon 25 Mar 2024 06:02:46 PM UTC. 2024-03-25T19:24:38.756 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-25T19:24:38.757 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:38.758 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.3 M 2024-03-25T19:24:38.758 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 5.5 M 2024-03-25T19:24:38.758 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-25T19:24:42.361 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 356 kB/s | 1.3 MB 00:03 2024-03-25T19:24:42.362 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:42.362 INFO:teuthology.orchestra.run.smithi029.stdout:Total 356 kB/s | 1.3 MB 00:03 2024-03-25T19:24:42.362 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-25T19:24:42.367 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-25T19:24:42.367 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-25T19:24:42.413 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-25T19:24:42.414 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-25T19:24:42.669 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-25T19:24:42.765 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:43.333 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:43.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:43.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:43.618 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-25T19:24:43.618 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:43.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:24:43.618 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-25T19:24:43.808 DEBUG:teuthology.parallel:result is None 2024-03-25T19:28:42.880 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 730 kB/s | 287 MB 06:42 2024-03-25T19:28:42.880 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:42.881 INFO:teuthology.orchestra.run.smithi196.stdout:Total 730 kB/s | 287 MB 06:42 2024-03-25T19:28:42.881 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:28:42.884 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:28:42.884 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:28:44.586 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:28:44.586 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:29:13.890 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:29:13.999 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:21.142 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:21.435 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:29:21.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:29:22.212 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:26:37 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:29:22.360 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:29:22.361 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:22.361 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:29:22.361 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:22.361 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:29:22.361 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 5.9 M 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 28 M 2024-03-25T19:29:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:29:31.677 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 653 kB/s | 5.9 MB 00:09 2024-03-25T19:29:31.678 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:31.678 INFO:teuthology.orchestra.run.smithi196.stdout:Total 653 kB/s | 5.9 MB 00:09 2024-03-25T19:29:31.679 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:29:31.683 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:29:31.683 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:29:31.756 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:29:31.756 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:29:32.502 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:29:32.609 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:29:33.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:29:34.164 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:26:49 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:29:34.315 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:29:34.315 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:29:34.316 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:34.317 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 109 k 2024-03-25T19:29:34.317 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 440 k 2024-03-25T19:29:34.317 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:29:34.913 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 182 kB/s | 109 kB 00:00 2024-03-25T19:29:34.914 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:34.914 INFO:teuthology.orchestra.run.smithi196.stdout:Total 182 kB/s | 109 kB 00:00 2024-03-25T19:29:34.914 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:29:34.918 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:29:34.918 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:29:34.952 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:29:34.952 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:29:35.072 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:29:35.186 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:29:35.556 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:29:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:29:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:35.834 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:29:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:35.834 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:29:36.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:29:36.572 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:26:51 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:29:36.720 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:29:36.721 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:36.722 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:29:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:36.722 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 5.4 M 2024-03-25T19:29:36.722 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 25 M 2024-03-25T19:29:36.722 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:29:46.198 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 578 kB/s | 5.4 MB 00:09 2024-03-25T19:29:46.199 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:46.199 INFO:teuthology.orchestra.run.smithi196.stdout:Total 578 kB/s | 5.4 MB 00:09 2024-03-25T19:29:46.199 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:29:46.204 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:29:46.204 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:29:46.272 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:29:46.272 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:29:46.940 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:29:47.047 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:47.831 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:29:48.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2-debuginfo 2024-03-25T19:29:48.629 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:27:03 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:29:48.777 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:29:48.777 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:29:48.778 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:29:48.779 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 44 M 2024-03-25T19:29:48.779 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 253 M 2024-03-25T19:29:48.779 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:30:42.735 INFO:teuthology.orchestra.run.smithi196.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 840 kB/s | 44 MB 00:53 2024-03-25T19:30:42.736 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:42.736 INFO:teuthology.orchestra.run.smithi196.stdout:Total 840 kB/s | 44 MB 00:53 2024-03-25T19:30:42.736 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:30:42.740 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:30:42.740 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:30:43.039 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:30:43.039 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:30:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:30:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:30:49.990 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:30:50.270 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:30:50.271 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:30:50.271 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:30:50.271 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:50.271 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:30:50.271 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:30:50.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:30:51.026 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:28:06 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:30:51.175 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:30:51.175 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 52 M 2024-03-25T19:30:51.176 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 308 M 2024-03-25T19:30:51.177 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:31:46.003 INFO:teuthology.orchestra.run.smithi196.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 968 kB/s | 52 MB 00:54 2024-03-25T19:31:46.004 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:46.004 INFO:teuthology.orchestra.run.smithi196.stdout:Total 968 kB/s | 52 MB 00:54 2024-03-25T19:31:46.004 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:31:46.008 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:31:46.008 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:31:46.351 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:31:46.352 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:31:52.365 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:31:52.476 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:31:54.566 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:54.837 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:31:55.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:31:55.607 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:10 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:31:55.756 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:55.757 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:31:55.758 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:55.758 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 615 k 2024-03-25T19:31:55.758 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.1 M 2024-03-25T19:31:55.758 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:31:56.647 INFO:teuthology.orchestra.run.smithi196.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 692 kB/s | 615 kB 00:00 2024-03-25T19:31:56.647 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:56.648 INFO:teuthology.orchestra.run.smithi196.stdout:Total 691 kB/s | 615 kB 00:00 2024-03-25T19:31:56.648 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:31:56.651 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:31:56.651 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:31:56.690 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:31:56.691 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:31:56.854 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:31:56.971 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:57.356 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:57.634 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:31:57.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:31:58.406 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:13 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:31:58.558 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 771 k 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.7 M 2024-03-25T19:31:58.559 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:00.591 INFO:teuthology.orchestra.run.smithi196.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 379 kB/s | 771 kB 00:02 2024-03-25T19:32:00.592 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:00.592 INFO:teuthology.orchestra.run.smithi196.stdout:Total 379 kB/s | 771 kB 00:02 2024-03-25T19:32:00.592 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:00.596 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:00.596 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:00.639 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:00.639 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:00.820 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:00.923 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:32:01.331 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:32:01.605 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:32:01.605 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:01.606 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:01.606 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:01.606 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:01.606 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:01.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:32:02.383 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:17 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:32:02.531 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:02.532 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 1.1 M 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 4.3 M 2024-03-25T19:32:02.533 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:03.844 INFO:teuthology.orchestra.run.smithi196.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 878 kB/s | 1.1 MB 00:01 2024-03-25T19:32:03.845 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:03.845 INFO:teuthology.orchestra.run.smithi196.stdout:Total 877 kB/s | 1.1 MB 00:01 2024-03-25T19:32:03.846 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:03.849 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:03.849 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:03.894 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:03.895 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:04.136 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:04.235 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:04.602 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:04.871 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:04.872 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:04.872 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:04.872 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:04.872 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:04.872 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:05.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:32:05.645 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:20 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:32:05.793 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:32:05.793 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:32:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:05.795 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 337 k 2024-03-25T19:32:05.795 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.0 M 2024-03-25T19:32:05.795 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:06.655 INFO:teuthology.orchestra.run.smithi196.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 392 kB/s | 337 kB 00:00 2024-03-25T19:32:06.656 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:06.656 INFO:teuthology.orchestra.run.smithi196.stdout:Total 391 kB/s | 337 kB 00:00 2024-03-25T19:32:06.656 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:06.660 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:06.660 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:06.697 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:06.698 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:06.834 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:06.938 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:07.587 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:07.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:32:08.364 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:23 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:32:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:32:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:08.512 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:32:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 509 k 2024-03-25T19:32:08.513 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.5 M 2024-03-25T19:32:08.514 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:09.625 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 458 kB/s | 509 kB 00:01 2024-03-25T19:32:09.626 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:09.626 INFO:teuthology.orchestra.run.smithi196.stdout:Total 457 kB/s | 509 kB 00:01 2024-03-25T19:32:09.626 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:09.631 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:09.631 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:09.671 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:09.672 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:09.832 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:09.958 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:10.313 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:10.579 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:10.579 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:10.579 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:10.579 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:10.580 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:10.580 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:10.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:32:11.321 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:29:26 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:32:11.473 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:32:11.473 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:11.473 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:32:11.473 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 52 M 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 298 M 2024-03-25T19:32:11.474 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:43.648 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.6 MB/s | 52 MB 00:32 2024-03-25T19:32:43.650 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:43.650 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.6 MB/s | 52 MB 00:32 2024-03-25T19:32:43.650 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:43.653 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:43.653 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:43.987 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:43.987 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:49.996 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:51.973 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:52.256 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:52.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:32:53.020 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 1:30:08 ago on Mon 25 Mar 2024 06:02:45 PM UTC. 2024-03-25T19:32:53.171 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 1.3 M 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 5.5 M 2024-03-25T19:32:53.173 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-25T19:32:54.275 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:32:54.275 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:54.276 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:32:54.276 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-25T19:32:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-25T19:32:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-25T19:32:54.325 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-25T19:32:54.325 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-25T19:32:54.571 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-25T19:32:54.673 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:32:55.133 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-25T19:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-25T19:32:55.629 DEBUG:teuthology.parallel:result is None 2024-03-25T19:32:55.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:32:55.857 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:32:55.881 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:32:55.882 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:32:55.882 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:32:55.884 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:32:56.022 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:32:56.043 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:32:56.043 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:32:56.043 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:32:56.045 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:32:56.045 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:56.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:32:56.076 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:56.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:32:56.110 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:32:56.111 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:56.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:32:56.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:32:56.217 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:56.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:32:56.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:32:56.311 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:32:56.312 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:56.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:32:56.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:32:56.409 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:56.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:32:56.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:32:56.502 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:32:56.502 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:56.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:32:56.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:32:56.600 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:56.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:32:56.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:32:56.693 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:32:56.776 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:32:56.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-03-25T19:32:56.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-03-25T19:32:56.805 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:32:56.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:32:56.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:32:56.874 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:32:56.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:32:56.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:32:56.953 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:32:56.953 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:32:56.953 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621318', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDRDJbZDpFbLIzdGv/r9q9qfwyAtOCD9AXuxprSspTsrYGvfwm7W3TqHygDf1BCqYWLQ3OzFfScuo2cPStyYEQ=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBTCwSZrmGLv4xlFHncf1n8NvCYQh1zPjWy/plhZ2/6WW1z/UzXC2gWa2V0i5EyKIgN62bv3j0SjgFo+YQo7vQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:32:56.953 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:32:56.985 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:32:56.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:32:57.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:32:57.066 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:57.066 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:32:57.121 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:32:57.121 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-03-25T19:32:57.176 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:32:57.176 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:57.176 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:32:57.176 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:57.176 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:57.177 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-25 18:10:27.645868564 +0000 2024-03-25T19:32:57.177 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-25 18:01:39.148324762 +0000 2024-03-25T19:32:57.177 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-25 18:01:39.148324762 +0000 2024-03-25T19:32:57.177 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-25 18:01:39.148324762 +0000 2024-03-25T19:32:57.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:32:57.242 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-25T19:32:57.242 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-25T19:32:57.242 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.8946e-05 s, 5.2 MB/s 2024-03-25T19:32:57.243 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:32:57.299 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-25 18:10:27.645868564 +0000 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-25 18:01:39.486319002 +0000 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-25 18:01:39.486319002 +0000 2024-03-25T19:32:57.354 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-25 18:01:39.486319002 +0000 2024-03-25T19:32:57.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:32:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-25T19:32:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-25T19:32:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.2197e-05 s, 6.2 MB/s 2024-03-25T19:32:57.421 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:32:57.476 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-25 18:10:27.646868547 +0000 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-25 18:01:39.813313429 +0000 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-25 18:01:39.813313429 +0000 2024-03-25T19:32:57.532 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-25 18:01:39.813313429 +0000 2024-03-25T19:32:57.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:32:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-25T19:32:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-25T19:32:57.598 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.0544e-05 s, 5.7 MB/s 2024-03-25T19:32:57.599 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:32:57.654 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-03-25T19:32:57.708 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:32:57.708 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-25 18:10:27.646868547 +0000 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-25 18:01:40.145307771 +0000 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-25 18:01:40.145307771 +0000 2024-03-25T19:32:57.709 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-25 18:01:40.145307771 +0000 2024-03-25T19:32:57.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:32:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-25T19:32:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-25T19:32:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.8761e-05 s, 5.8 MB/s 2024-03-25T19:32:57.774 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:32:57.829 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:32:57.829 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:57.829 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:32:57.848 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:32:57.848 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-25 18:10:05.893058384 +0000 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-25 18:01:23.600595458 +0000 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-25 18:01:23.600595458 +0000 2024-03-25T19:32:57.906 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-25 18:01:23.600595458 +0000 2024-03-25T19:32:57.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:32:57.976 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-25T19:32:57.976 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-25T19:32:57.976 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000223897 s, 2.3 MB/s 2024-03-25T19:32:57.977 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:32:58.036 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-25T19:32:58.094 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:32:58.094 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:58.094 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-25 18:10:05.893058384 +0000 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-25 18:01:23.937588103 +0000 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-25 18:01:23.937588103 +0000 2024-03-25T19:32:58.095 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-25 18:01:23.937588103 +0000 2024-03-25T19:32:58.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:32:58.165 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-25T19:32:58.165 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-25T19:32:58.165 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000247414 s, 2.1 MB/s 2024-03-25T19:32:58.166 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:32:58.225 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-25T19:32:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:32:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:58.281 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-25 18:10:05.894058363 +0000 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-25 18:01:24.283580551 +0000 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-25 18:01:24.283580551 +0000 2024-03-25T19:32:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-25 18:01:24.283580551 +0000 2024-03-25T19:32:58.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:32:58.350 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-25T19:32:58.350 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-25T19:32:58.350 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000264035 s, 1.9 MB/s 2024-03-25T19:32:58.351 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:32:58.408 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-25 18:10:05.894058363 +0000 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-25 18:01:24.604573545 +0000 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-25 18:01:24.604573545 +0000 2024-03-25T19:32:58.464 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-25 18:01:24.604573545 +0000 2024-03-25T19:32:58.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:32:58.531 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-25T19:32:58.532 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-25T19:32:58.532 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000200452 s, 2.6 MB/s 2024-03-25T19:32:58.533 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:32:58.590 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:32:58.590 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:32:58.590 INFO:tasks.ceph:Generating config... 2024-03-25T19:32:58.591 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:32:58.591 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:32:58.591 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:32:58.591 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:32:58.591 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:32:58.591 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:32:58.591 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:32:58.591 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:32:58.592 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:32:58.592 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:32:58.592 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:32:58.592 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:32:58.593 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:32:58.594 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:32:58.594 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:32:58.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:32:58.651 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:32:58.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:32:58.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:32:58.726 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.29:6789]'), ('mon.c', '[v1:172.21.15.29:6790]'), ('mon.b', '[v1:172.21.15.196:6789]')] 2024-03-25T19:32:58.727 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.29:6789],[v1:172.21.15.29:6790],[v1:172.21.15.196:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:32:58.727 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:58.728 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:32:58.781 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.29:6789]' --addv c '[v1:172.21.15.29:6790]' --addv b '[v1:172.21.15.196:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:32:58.867 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 3 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:32:58.868 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:32:58.869 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:32:58.883 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 82fd6393-df4e-4dec-8d3f-095400b97e2d... 2024-03-25T19:32:58.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:32:58.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:32:58.974 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-03-25T19:32:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-03-25T19:32:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:32:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2024-03-25T19:32:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:32:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:32:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.29:6790],[v1:172.21.15.196:6789]" 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: mon election default strategy = 3 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: ms bind msgr2 = False 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 5000 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: ms type = async 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: osd heartbeat grace = 80 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:32:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-03-25T19:32:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-03-25T19:32:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bdev async discard = True 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bdev enable discard = True 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = stupid 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore zero block detection = True 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1 2024-03-25T19:32:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd class default list = * 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd class load list = * 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd client watch timeout = 120 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify cached snaps = True 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify missing on start = True 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:32:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: osd fast shutdown = False 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd max markdown count = 1000 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock override recovery settings = True 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue = debug_random 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue cut off = debug_random 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: osd shutdown pgref assert = True 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-03-25T19:32:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-03-25T19:32:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:32:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: mon min osdmap epochs = 50 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd crush smoke test = False 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune interval = 2 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune min = 15 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon scrub interval = 300 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on pool no app = False 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: paxos service trim min = 10 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2024-03-25T19:32:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: debug objecter = 20 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: debug rados = 20 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:32:58.986 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-03-25T19:32:58.987 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:32:58.988 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.29:6790],[v1:172.21.15.196:6789]" 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: mon election default strategy = 3 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: ms bind msgr2 = False 2024-03-25T19:32:58.989 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: ms type = async 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd heartbeat grace = 80 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-03-25T19:32:58.990 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.991 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bdev async discard = True 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bdev enable discard = True 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore allocator = stupid 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore zero block detection = True 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 1 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 1 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-03-25T19:32:58.992 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd class default list = * 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd class load list = * 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd client watch timeout = 120 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify cached snaps = True 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify missing on start = True 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd fast shutdown = False 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd max markdown count = 1000 2024-03-25T19:32:58.993 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock override recovery settings = True 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue = debug_random 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue cut off = debug_random 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-25T19:32:58.994 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:32:58.995 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon min osdmap epochs = 50 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd crush smoke test = False 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune interval = 2 2024-03-25T19:32:58.996 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune min = 15 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: mon scrub interval = 300 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on pool no app = False 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: paxos service trim min = 10 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: debug objecter = 20 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout: debug rados = 20 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-03-25T19:32:58.997 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-03-25T19:32:58.998 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-03-25T19:32:58.999 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:32:58.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:32:59.053 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:32:59.053 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:59.054 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:32:59.068 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:59.068 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:32:59.122 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:32:59.122 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:59.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:32:59.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:32:59.200 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:32:59.201 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:32:59.254 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:32:59.255 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:59.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:32:59.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:32:59.297 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:32:59.298 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:32:59.353 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:32:59.353 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:32:59.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:32:59.407 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:32:59.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:32:59.478 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:32:59.481 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:32:59.481 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:32:59.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:32:59.523 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:32:59.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:32:59.581 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:32:59.596 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:32:59.597 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:32:59.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:32:59.623 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:32:59.623 INFO:tasks.ceph:role: osd.0 2024-03-25T19:32:59.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:32:59.623 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:32:59.689 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:32:59.721 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-03-25T19:32:59.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-03-25T19:32:59.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:32:59.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:32:59.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:32:59.931 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:32:59.932 INFO:tasks.ceph:role: osd.1 2024-03-25T19:32:59.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:32:59.932 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:32:59.998 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:32:59.999 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:00.031 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-03-25T19:33:00.033 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-03-25T19:33:00.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:00.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:00.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:00.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:00.243 INFO:tasks.ceph:role: osd.2 2024-03-25T19:33:00.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:33:00.243 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:33:00.309 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:00.309 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:00.309 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:00.309 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:00.309 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:00.310 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:00.310 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:00.310 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:00.310 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:00.310 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:00.342 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-03-25T19:33:00.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-03-25T19:33:00.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:00.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:00.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:00.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:00.551 INFO:tasks.ceph:role: osd.3 2024-03-25T19:33:00.551 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:33:00.551 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:00.617 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:00.618 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:00.618 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:00.618 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:00.618 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:00.649 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-03-25T19:33:00.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-03-25T19:33:00.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:00.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:00.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.893+0000 7f3aea8d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.893+0000 7f3aea8d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.893+0000 7f3aea8d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.893+0000 7f3aea8d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.893+0000 7f3aea8d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:00.894+0000 7f3aea8d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:33:03.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:33:03.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:03.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:03.852+0000 7fb512149640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:33:06.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:06.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.816+0000 7fb2cc111640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.816+0000 7fb2cc111640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.816+0000 7fb2cc111640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.816+0000 7fb2cc111640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.816+0000 7fb2cc111640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:06.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:06.817+0000 7fb2cc111640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:33:09.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:09.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.778+0000 7fb64460b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.778+0000 7fb64460b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.778+0000 7fb64460b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.778+0000 7fb64460b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.778+0000 7fb64460b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:33:09.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:09.779+0000 7fb64460b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:33:12.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:12.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:33:12.687 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:12.687 INFO:tasks.ceph:role: osd.4 2024-03-25T19:33:12.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:33:12.687 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:33:12.764 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:12.765 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:13.269 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-25T19:33:13.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-25T19:33:13.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:33:13.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:33:13.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:33:13.505 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:13.506 INFO:tasks.ceph:role: osd.5 2024-03-25T19:33:13.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:33:13.506 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:33:13.580 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:13.580 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:13.580 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:14.040 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-25T19:33:14.042 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-25T19:33:14.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:33:14.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:33:14.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:33:14.236 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:14.236 INFO:tasks.ceph:role: osd.6 2024-03-25T19:33:14.236 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:33:14.236 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:33:14.311 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:14.312 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:14.818 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-25T19:33:14.820 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-25T19:33:14.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:33:14.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:33:15.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:33:15.075 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:15.076 INFO:tasks.ceph:role: osd.7 2024-03-25T19:33:15.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:33:15.076 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:15.152 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:15.153 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:15.153 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:15.660 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-25T19:33:15.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-25T19:33:15.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:33:15.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:33:15.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:15.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.928+0000 7f46c4fdf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:33:15.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.928+0000 7f46c4fdf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:33:15.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.928+0000 7f46c4fdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:33:15.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.928+0000 7f46c4fdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:33:15.928 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.928+0000 7f46c4fdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:33:15.929 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:15.929+0000 7f46c4fdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:33:18.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:33:18.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:18.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:33:18.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:33:18.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:33:18.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:33:18.889 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:33:18.890 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:18.889+0000 7fb7044ce640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:33:21.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:33:21.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:21.853 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.853+0000 7feb6ad1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:33:21.853 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.854+0000 7feb6ad1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:33:21.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.854+0000 7feb6ad1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:33:21.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.854+0000 7feb6ad1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:33:21.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.854+0000 7feb6ad1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:33:21.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:21.854+0000 7feb6ad1d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:33:24.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:33:24.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:24.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.820+0000 7f1db1c1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:33:24.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.820+0000 7f1db1c1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:33:24.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.820+0000 7f1db1c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:33:24.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.821+0000 7f1db1c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:33:24.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.821+0000 7f1db1c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:33:24.822 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:24.821+0000 7f1db1c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:33:27.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:33:27.691 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:33:27.691 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:27.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:33:27.723 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:27.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:33:27.794 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:27.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:33:27.865 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:27.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:33:27.935 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:27.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:33:28.004 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:33:28.036 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:33:28.108 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:33:28.178 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:33:28.249 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:33:28.318 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:28.318 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:33:28.335 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:28.335 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:33:28.374 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:33:28.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:33:28.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:33:28.417 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBr0QFmigBjGBAAgtKv6wf8r5gjfdvYFJ+hkw== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBs0QFmc4JRNRAArmFbS1DKcWb2hRPx2OXVbw== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBv0QFmZN3YMhAAG7I5wM+uwY98bGWmhr5Tkg== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBy0QFmwkq3MBAAvKUA3n0V48zvhafxy6KFGw== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB10QFm0kRzLhAAfqhka6dl6yQZT2GogjH/Ug== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBr0QFmsx2fHBAA+u1+kRfK6oW0i5H1MOnokg== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB70QFmxUNnNxAALTuwhNQEmkt5OWQL8kibCQ== 2024-03-25T19:33:28.418 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB+0QFmRmsMNRAAmktfBUg9oVz0TtqZzoIe+A== 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCB0QFmFTvzMhAAk8MNmvit5H8AbU/PezPrlQ== 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCE0QFmuukIMRAAx6qUDLqq1LB44ICrCIpr+A== 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBr0QFmaQ5HHxAAADaZl4ev8D8t713gBqe3bg== 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2024-03-25T19:33:28.419 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBr0QFm1uq/IhAAIKeeiR2QmenoEEzZCEGjUg== 2024-03-25T19:33:28.441 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2024-03-25T19:33:28.441 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBr0QFmigBjGBAAgtKv6wf8r5gjfdvYFJ+hkw== 2024-03-25T19:33:28.441 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-03-25T19:33:28.441 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBs0QFmc4JRNRAArmFbS1DKcWb2hRPx2OXVbw== 2024-03-25T19:33:28.441 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBv0QFmZN3YMhAAG7I5wM+uwY98bGWmhr5Tkg== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBy0QFmwkq3MBAAvKUA3n0V48zvhafxy6KFGw== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQB10QFm0kRzLhAAfqhka6dl6yQZT2GogjH/Ug== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBr0QFmsx2fHBAA+u1+kRfK6oW0i5H1MOnokg== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQB70QFmxUNnNxAALTuwhNQEmkt5OWQL8kibCQ== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQB+0QFmRmsMNRAAmktfBUg9oVz0TtqZzoIe+A== 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2024-03-25T19:33:28.442 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCB0QFmFTvzMhAAk8MNmvit5H8AbU/PezPrlQ== 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCE0QFmuukIMRAAx6qUDLqq1LB44ICrCIpr+A== 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBr0QFmaQ5HHxAAADaZl4ev8D8t713gBqe3bg== 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-03-25T19:33:28.443 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBr0QFm1uq/IhAAIKeeiR2QmenoEEzZCEGjUg== 2024-03-25T19:33:28.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:33:28.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:33:28.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:33:28.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:33:28.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:28.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:33:29.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:33:29.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:33:29.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:33:29.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:33:29.133 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:33:29.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:33:29.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:33:29.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:33:29.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:33:29.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:33:29.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:33:29.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:33:29.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:33:29.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:33:29.627 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:29.629 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:29.680 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:33:29.681 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:33:29.681 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:33:29.681 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:33:29.683 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:33:29.683 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:33:29.683 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:33:29.684 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:33:29.685 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:33:29.685 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:33:29.686 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:33:29.686 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:33:29.723 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:33:29.723 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:33:29.723 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:33:29.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:33:29.725 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:33:29.726 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:33:29.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:33:29.728 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:33:29.728 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:33:29.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.829+0000 7fa7411af640 10 monclient: get_monmap_and_config 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.829+0000 7fa7411af640 10 monclient: build_initial_monmap 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.829+0000 7fa7411af640 10 monclient: monmap: 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:33:29.831419+0000 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:33:29.831419+0000 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.830+0000 7fa7411af640 1 Processor -- start 2024-03-25T19:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.830+0000 7fa7411af640 1 -- start start 2024-03-25T19:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.830+0000 7fa7411af640 10 monclient: init 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-a con 0x7fa73c142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-b con 0x7fa73c0a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa73c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0a9d60 con 0x7fa73c142a20 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa73c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c13ff10 con 0x7fa73c0b4700 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0aa1f0 con 0x7fa73c0a96b0 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:29.831+0000 7fa7411af640 10 monclient(hunting): authenticate will time out at 6086.585148s 2024-03-25T19:33:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.032+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa73c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.032+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa73c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.032+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:30.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.433+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa73c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:30.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.433+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa73c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:30.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:30.433+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:31.234+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa73c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:31.234+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa73c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:31.234+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa73c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa73c13f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa73c0a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa73c142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0b4ae0 con 0x7fa73c142a20 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0a9d60 con 0x7fa73c0b4700 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c13ff10 con 0x7fa73c0a96b0 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:32.831+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.032+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.032+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.032+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.433+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.433+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:33.433+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:34.234+0000 7fa73b59e640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:34.234+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:34.234+0000 7fa73ad9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800c730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa73c142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa73c0a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0aa1f0 con 0x7fa73c142a20 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0b4ae0 con 0x7fa73c0b4700 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0a9d60 con 0x7fa73c0a96b0 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa73ad9d640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:35.831+0000 7fa73b59e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:36.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.032+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:36.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.032+0000 7fa73ad9d640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:36.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.032+0000 7fa73b59e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.433+0000 7fa73ad9d640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.433+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:36.433+0000 7fa73b59e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:36.744 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:00:06.906 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.050 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:00:07.212 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:37.234+0000 7fa73ad9d640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2024-03-25T19:33:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:37.234+0000 7fa73a59c640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6790/0 2024-03-25T19:33:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:37.234+0000 7fa73b59e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-03-25T19:33:37.246 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:00:07.410 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.247 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:00:07.411 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.574 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:00:07.738 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.575 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:00:07.738 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa7280056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:38.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa73c0a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa73c142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c13ff10 con 0x7fa73c142a20 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0aa1f0 con 0x7fa73c0b4700 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0b4ae0 con 0x7fa73c0a96b0 2024-03-25T19:33:38.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.832+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.888+0000 7fa73a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 0x7fa728008ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:53816/0 (socket says 172.21.15.29:53816) 2024-03-25T19:33:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:38.888+0000 7fa73a59c640 1 -- 172.21.15.29:0/2253644255 learned_addr learned my addr 172.21.15.29:0/2253644255 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.276+0000 7fa73ad9d640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa72800c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T19:33:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.276+0000 7fa73ad9d640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa72800c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:33:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.276+0000 7fa73ad9d640 1 --1- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 0x7fa72800c720 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:33:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.310+0000 7fa73b59e640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:33:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.310+0000 7fa73b59e640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:33:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:39.310+0000 7fa73b59e640 1 --1- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 0x7fa72800ffa0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa73c0a96b0 legacy=0x7fa72800c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa73c142a20 legacy=0x7fa72800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa728005780 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa72800c720 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0a9d60 con 0x7fa72800c720 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa728005c40 con 0x7fa73c0b4700 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72800cba0 con 0x7fa728005780 2024-03-25T19:33:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:41.832+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:44.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa728005780 legacy=0x7fa728010770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa72800c720 legacy=0x7fa7280095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa728010170 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa728005780 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72800ffa0 con 0x7fa728010170 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280095d0 con 0x7fa73c0b4700 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa728009760 con 0x7fa728005780 2024-03-25T19:33:44.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:44.833+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): tick 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa728005780 legacy=0x7fa728010770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa728008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa728010170 legacy=0x7fa728009ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa72800a6a0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa728005780 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa728009940 con 0x7fa72800a6a0 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72800c1a0 con 0x7fa73c0b4700 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72800c3d0 con 0x7fa728005780 2024-03-25T19:33:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:47.833+0000 7fa738d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382661107 0 0) 0x7fa72800c3d0 con 0x7fa728005780 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:33:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.079+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa724003520 con 0x7fa728005780 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.090+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932629461 0 0) 0x7fa72800c1a0 con 0x7fa73c0b4700 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.090+0000 7fa739d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.090+0000 7fa739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.090+0000 7fa739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.090+0000 7fa739d9b640 10 monclient(hunting): my global_id is 4102 2024-03-25T19:33:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.091+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa72800c3d0 con 0x7fa73c0b4700 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315845909 0 0) 0x7fa728009940 con 0x7fa72800a6a0 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:33:48.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.096+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa72800c1a0 con 0x7fa72800a6a0 2024-03-25T19:33:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159074718 0 0) 0x7fa72800c1a0 con 0x7fa72800a6a0 2024-03-25T19:33:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6790/0 conn(0x7fa728005780 legacy=0x7fa728010770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa728009c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 monclient: found mon.noname-a 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa728008eb0 con 0x7fa72800a6a0 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa72c003f20 con 0x7fa72800a6a0 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa739d9b640 10 monclient: dump: 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:48.196 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa7411af640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:33:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.194+0000 7fa7411af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa72c004080 con 0x7fa72800a6a0 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 1 -- 172.21.15.29:0/2253644255 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa72c0044d0 con 0x7fa72800a6a0 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa739d9b640 10 monclient: dump: 2024-03-25T19:33:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 4 set_mon_vals no callback set 2024-03-25T19:33:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 10 monclient: shutdown 2024-03-25T19:33:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 1 -- 172.21.15.29:0/2253644255 >> v1:172.21.15.29:6789/0 conn(0x7fa72800a6a0 legacy=0x7fa72800aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 1 -- 172.21.15.29:0/2253644255 shutdown_connections 2024-03-25T19:33:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 1 -- 172.21.15.29:0/2253644255 >> 172.21.15.29:0/2253644255 conn(0x7fa73c0668e0 msgr2=0x7fa73c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 1 -- 172.21.15.29:0/2253644255 shutdown_connections 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.219+0000 7fa7411af640 1 -- 172.21.15.29:0/2253644255 wait complete. 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.220+0000 7fa7411af640 10 monclient: build_initial_monmap 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.220+0000 7fa7411af640 10 monclient: monmap: 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 librados: starting msgr at 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 librados: starting objecter 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 Processor -- start 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 -- start start 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 librados: setting wanted keys 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 1 librados: calling monclient init 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 10 monclient: init 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-a con 0x7fa728005780 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.221+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-c con 0x7fa72800a6a0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 10 monclient(hunting): picked mon.noname-b con 0x7fa73c0b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0b1970 con 0x7fa728005780 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c1ca1c0 con 0x7fa73c0b4700 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c1ca4a0 con 0x7fa72800a6a0 2024-03-25T19:33:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7411af640 10 monclient(hunting): authenticate will time out at 6104.974892s 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa73a59c640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fa72800a6a0 0x7fa73c13b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:49564/0 (socket says 172.21.15.29:49564) 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa73a59c640 1 -- 172.21.15.29:0/532263533 learned_addr learned my addr 172.21.15.29:0/532263533 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7237fe640 10 client.?.objecter ms_handle_connect 0x7fa72800a6a0 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.222+0000 7fa7237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.224+0000 7fa7237fe640 10 client.?.objecter ms_handle_connect 0x7fa73c0b4700 2024-03-25T19:33:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.224+0000 7fa7237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.280+0000 7fa7237fe640 10 client.?.objecter ms_handle_connect 0x7fa728005780 2024-03-25T19:33:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.280+0000 7fa7237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237185030 0 0) 0x7fa73c0b1970 con 0x7fa728005780 2024-03-25T19:33:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:33:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.299+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa728009940 con 0x7fa728005780 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456412464 0 0) 0x7fa73c1ca4a0 con 0x7fa72800a6a0 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa73c0b1970 con 0x7fa72800a6a0 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635713213 0 0) 0x7fa728009940 con 0x7fa728005780 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa73c1ca4a0 con 0x7fa728005780 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa72c004450 con 0x7fa728005780 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): dump: 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.305+0000 7fa7237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1292772972 0 0) 0x7fa73c1ca4a0 con 0x7fa728005780 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 >> v1:172.21.15.29:6790/0 conn(0x7fa72800a6a0 legacy=0x7fa73c13b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 >> v1:172.21.15.196:6789/0 conn(0x7fa73c0b4700 legacy=0x7fa73c0aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 1 monclient: found mon.a 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c1c8420 con 0x7fa728005780 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 10 monclient: _renew_subs 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73c13b8f0 con 0x7fa728005780 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 10 monclient: _renew_subs 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73c0af880 con 0x7fa728005780 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.311+0000 7fa7411af640 1 librados: init done 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.312+0000 7fa7411af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.312+0000 7fa7411af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.312+0000 7fa7411af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.312+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73c0afb30 con 0x7fa728005780 2024-03-25T19:33:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa72c004860 con 0x7fa728005780 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa72c004c70 con 0x7fa728005780 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa7237fe640 10 monclient: dump: 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa72c005220 con 0x7fa728005780 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.484+0000 7fa73959a640 4 set_mon_vals no callback set 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2033900722 0 0) 0x7fa72c006690 con 0x7fa728005780 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 10 client.4113.objecter ms_dispatch 0x7fa73c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa72c006900 con 0x7fa728005780 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.485+0000 7fa7237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.554+0000 7fa7411af640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:33:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.554+0000 7fa7411af640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:33:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.554+0000 7fa7411af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.554+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa73c0afb30 con 0x7fa728005780 2024-03-25T19:33:48.713 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:48.713+0000 7f9936441040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:33:48.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.750+0000 7fa7237fe640 1 -- 172.21.15.29:0/532263533 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fa72c0055b0 con 0x7fa728005780 2024-03-25T19:33:48.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.750+0000 7fa7237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:33:48.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.750+0000 7fa7237fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:33:48.750 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 10 librados: watch_flush enter 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 10 librados: watch_flush exit 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 10 monclient: shutdown 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 >> v1:172.21.15.29:6789/0 conn(0x7fa728005780 legacy=0x7fa73c1c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 shutdown_connections 2024-03-25T19:33:48.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 >> 172.21.15.29:0/532263533 conn(0x7fa73c0668e0 msgr2=0x7fa73c13d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 shutdown_connections 2024-03-25T19:33:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 -- 172.21.15.29:0/532263533 wait complete. 2024-03-25T19:33:48.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:48.751+0000 7fa7411af640 1 librados: shutdown 2024-03-25T19:33:48.752 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:48.751+0000 7f3ab5e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:33:48.761 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:33:48.761 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:33:48.761 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:48.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:33:48.770 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:48.770+0000 7f9936441040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:33:48.789 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:48.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:33:48.828 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:48.828+0000 7f3ab5e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:33:48.854 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:48.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:33:48.921 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-25T19:33:48.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:33:48.988 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:48.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:33:49.016 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:49.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:33:49.083 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:49.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:33:49.151 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-25T19:33:49.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:33:49.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 5739ebf6-47a8-471b-bf92-123ad1d3f33c 0 2024-03-25T19:33:49.349 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.349+0000 7f9936441040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:33:49.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.359+0000 7f4969c93640 10 monclient: get_monmap_and_config 2024-03-25T19:33:49.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.359+0000 7f4969c93640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.359+0000 7f4969c93640 10 monclient: monmap: 2024-03-25T19:33:49.359 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:49.360932+0000 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:49.360932+0000 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:49.360 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.361 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.360+0000 7f4969c93640 1 Processor -- start 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.360+0000 7f4969c93640 1 -- start start 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.360+0000 7f4969c93640 10 monclient: init 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-a con 0x7f49640b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-b con 0x7f4964142a20 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-c con 0x7f49640a96b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640a9d60 con 0x7f49640b4700 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496413ff10 con 0x7f49640a96b0 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640aa1f0 con 0x7f4964142a20 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4969c93640 10 monclient(hunting): authenticate will time out at 6116.648954s 2024-03-25T19:33:49.362 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4963fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f49640a96b0 0x7f496413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51232/0 (socket says 172.21.15.196:51232) 2024-03-25T19:33:49.363 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.361+0000 7f4963fff640 1 -- 172.21.15.196:0/3025159755 learned_addr learned my addr 172.21.15.196:0/3025159755 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:49.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850556513 0 0) 0x7f49640aa1f0 con 0x7f4964142a20 2024-03-25T19:33:49.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.364 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.364+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c003520 con 0x7f4964142a20 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939915493 0 0) 0x7f496413ff10 con 0x7f49640a96b0 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:33:49.365 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.365+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49640aa1f0 con 0x7f49640a96b0 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330138666 0 0) 0x7f49640a9d60 con 0x7f49640b4700 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:33:49.367 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.367+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496413ff10 con 0x7f49640b4700 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013155758 0 0) 0x7f496413ff10 con 0x7f49640b4700 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 >> v1:172.21.15.29:6790/0 conn(0x7f4964142a20 legacy=0x7f496413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 >> v1:172.21.15.196:6789/0 conn(0x7f49640a96b0 legacy=0x7f496413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 monclient: found mon.noname-a 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49640b4ae0 con 0x7f49640b4700 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4954001930 con 0x7f49640b4700 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f49627fc640 10 monclient: dump: 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.373 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f4969c93640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.372+0000 7f4969c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4954001a90 con 0x7f49640b4700 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 1 -- 172.21.15.196:0/3025159755 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4954001ee0 con 0x7f49640b4700 2024-03-25T19:33:49.374 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f4969c93640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f4969c93640 4 set_mon_vals no callback set 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.374+0000 7f49627fc640 10 monclient: dump: 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.375 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 10 monclient: shutdown 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 1 -- 172.21.15.196:0/3025159755 >> v1:172.21.15.29:6789/0 conn(0x7f49640b4700 legacy=0x7f4964142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 1 -- 172.21.15.196:0/3025159755 shutdown_connections 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 1 -- 172.21.15.196:0/3025159755 >> 172.21.15.196:0/3025159755 conn(0x7f49640668e0 msgr2=0x7f4964066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 1 -- 172.21.15.196:0/3025159755 shutdown_connections 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.375+0000 7f4969c93640 1 -- 172.21.15.196:0/3025159755 wait complete. 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 10 monclient: monmap: 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:49.378 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 librados: starting msgr at 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 librados: starting objecter 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 Processor -- start 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 -- start start 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 librados: setting wanted keys 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 1 librados: calling monclient init 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.376+0000 7f4969c93640 10 monclient: init 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:49.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-c con 0x7f49640a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-a con 0x7f49640b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient(hunting): picked mon.noname-b con 0x7f4964142a20 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49641376c0 con 0x7f49640b4700 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49641379a0 con 0x7f4964142a20 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964135c10 con 0x7f49640a96b0 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.377+0000 7f4969c93640 10 monclient(hunting): authenticate will time out at 6116.664906s 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.378+0000 7f4962ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49640b4700 0x7f4964137dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.196:40968/0 (socket says 172.21.15.196:40968) 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.378+0000 7f4962ffd640 1 -- 172.21.15.196:0/3987903195 learned_addr learned my addr 172.21.15.196:0/3987903195 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.378+0000 7f494bfff640 10 client.?.objecter ms_handle_connect 0x7f49640b4700 2024-03-25T19:33:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.378+0000 7f494bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664849148 0 0) 0x7f49641376c0 con 0x7f49640b4700 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:33:49.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.381+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4950004eb0 con 0x7f49640b4700 2024-03-25T19:33:49.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335006798 0 0) 0x7f4950004eb0 con 0x7f49640b4700 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49641376c0 con 0x7f49640b4700 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49580019b0 con 0x7f49640b4700 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 10 monclient(hunting): dump: 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.388 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.388+0000 7f494bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3243765384 0 0) 0x7f49641376c0 con 0x7f49640b4700 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 >> v1:172.21.15.29:6790/0 conn(0x7f49640a96b0 legacy=0x7f4964136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 >> v1:172.21.15.196:6789/0 conn(0x7f4964142a20 legacy=0x7f49641e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 1 monclient: found mon.a 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964135150 con 0x7f49640b4700 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 10 monclient: _finish_auth 0 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f494bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:33:49.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 10 monclient: _renew_subs 2024-03-25T19:33:49.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49641358c0 con 0x7f49640b4700 2024-03-25T19:33:49.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:49.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 10 monclient: _renew_subs 2024-03-25T19:33:49.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4964141d70 con 0x7f49640b4700 2024-03-25T19:33:49.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.393+0000 7f4969c93640 1 librados: init done 2024-03-25T19:33:49.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.394+0000 7f4969c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.394+0000 7f4969c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.394+0000 7f4969c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.394+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4964135da0 con 0x7f49640b4700 2024-03-25T19:33:49.420 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.420+0000 7f9936441040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:33:49.466 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.466+0000 7f9936441040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:33:49.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.478+0000 7f3ab5e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:33:49.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4958001b10 con 0x7f49640b4700 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4958003a60 con 0x7f49640b4700 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 10 monclient: dump: 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.519 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.520 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.520 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4958004010 con 0x7f49640b4700 2024-03-25T19:33:49.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.519+0000 7f4961ffb640 4 set_mon_vals no callback set 2024-03-25T19:33:49.520 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 413047588 0 0) 0x7f4958006660 con 0x7f49640b4700 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 10 client.4119.objecter ms_dispatch 0x7f4964000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f49580069b0 con 0x7f49640b4700 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f494bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:49.521 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.520+0000 7f9936441040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:33:49.533 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.534+0000 7f3ab5e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:33:49.564 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.564+0000 7f9936441040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:33:49.580 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.581+0000 7f3ab5e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:33:49.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.592+0000 7f4969c93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5739ebf6-47a8-471b-bf92-123ad1d3f33c", "id": 0}] 2024-03-25T19:33:49.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.592+0000 7f4969c93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5739ebf6-47a8-471b-bf92-123ad1d3f33c", "id": 0}] 2024-03-25T19:33:49.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.592+0000 7f4969c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.592+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5739ebf6-47a8-471b-bf92-123ad1d3f33c", "id": 0} v 0) v1 -- 0x7f49640aab00 con 0x7f49640b4700 2024-03-25T19:33:49.636 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.637+0000 7f3ab5e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:33:49.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.679+0000 7f494bfff640 1 -- 172.21.15.196:0/3987903195 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5739ebf6-47a8-471b-bf92-123ad1d3f33c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3831373819 0 732707515) 0x7f49580057c0 con 0x7f49640b4700 2024-03-25T19:33:49.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f494bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5739ebf6-47a8-471b-bf92-123ad1d3f33c", "id": 0}] 2024-03-25T19:33:49.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f494bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:49.680 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 10 librados: watch_flush enter 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 10 librados: watch_flush exit 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 10 monclient: shutdown 2024-03-25T19:33:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.680+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 >> v1:172.21.15.29:6789/0 conn(0x7f49640b4700 legacy=0x7f4964137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 shutdown_connections 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 >> 172.21.15.196:0/3987903195 conn(0x7f49640668e0 msgr2=0x7f49640b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 shutdown_connections 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f4969c93640 1 -- 172.21.15.196:0/3987903195 wait complete. 2024-03-25T19:33:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f4969c93640 1 librados: shutdown 2024-03-25T19:33:49.682 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.681+0000 7f9936441040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:33:49.682 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.682+0000 7f3ab5e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:33:49.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 91604cd3-71e6-4c46-ab34-7b0d787ad733 1 2024-03-25T19:33:49.760 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.760+0000 7f9936441040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.794+0000 7fd201938640 10 monclient: get_monmap_and_config 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.794+0000 7fd201938640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.794+0000 7fd201938640 10 monclient: monmap: 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:49.795742+0000 2024-03-25T19:33:49.794 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:49.795742+0000 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:49.795 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.795+0000 7fd201938640 1 Processor -- start 2024-03-25T19:33:49.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.795+0000 7fd201938640 1 -- start start 2024-03-25T19:33:49.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.795+0000 7fd201938640 10 monclient: init 2024-03-25T19:33:49.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient(hunting): picked mon.noname-c con 0x7fd1fc0a9920 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient(hunting): picked mon.noname-a con 0x7fd1fc13fce0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient(hunting): picked mon.noname-b con 0x7fd1fc142ee0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0aa430 con 0x7fd1fc13fce0 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0b27f0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0b88e0 con 0x7fd1fc142ee0 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd201938640 10 monclient(hunting): authenticate will time out at 6117.083714s 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd1f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1fc13fce0 0x7fd1fc1427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.196:40982/0 (socket says 172.21.15.196:40982) 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd1f3fff640 1 -- 172.21.15.196:0/2815798174 learned_addr learned my addr 172.21.15.196:0/2815798174 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:49.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.796+0000 7fd1fb7fe640 1 --1- 172.21.15.196:0/2815798174 >> v1:172.21.15.29:6790/0 conn(0x7fd1fc142ee0 0x7fd1fc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.196:52432/0 (socket says 172.21.15.196:52432) 2024-03-25T19:33:49.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.798+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911706776 0 0) 0x7fd1fc0b88e0 con 0x7fd1fc142ee0 2024-03-25T19:33:49.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e8003520 con 0x7fd1fc142ee0 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254882095 0 0) 0x7fd1e4001110 con 0x7fd1fc13fce0 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:33:49.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1fc0b88e0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236749405 0 0) 0x7fd1fc0b27f0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:33:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.799+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e4001110 con 0x7fd1fc0a9920 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3401456930 0 0) 0x7fd1fc0b88e0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 >> v1:172.21.15.29:6790/0 conn(0x7fd1fc142ee0 legacy=0x7fd1fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 >> v1:172.21.15.196:6789/0 conn(0x7fd1fc0a9920 legacy=0x7fd1fc13f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 monclient: found mon.noname-a 2024-03-25T19:33:49.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0b45e0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd1e4001660 con 0x7fd1fc13fce0 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd201938640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd201938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd1fa7fc640 10 monclient: dump: 2024-03-25T19:33:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.803+0000 7fd201938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:49.806 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.805+0000 7f3ab5e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:33:49.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1e40017c0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 1 -- 172.21.15.196:0/2815798174 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd1e4001bd0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd1fa7fc640 10 monclient: dump: 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd201938640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.805+0000 7fd201938640 4 set_mon_vals no callback set 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 10 monclient: shutdown 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 1 -- 172.21.15.196:0/2815798174 >> v1:172.21.15.29:6789/0 conn(0x7fd1fc13fce0 legacy=0x7fd1fc1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 1 -- 172.21.15.196:0/2815798174 shutdown_connections 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 1 -- 172.21.15.196:0/2815798174 >> 172.21.15.196:0/2815798174 conn(0x7fd1fc0b7df0 msgr2=0x7fd1fc0b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 1 -- 172.21.15.196:0/2815798174 shutdown_connections 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 1 -- 172.21.15.196:0/2815798174 wait complete. 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.806+0000 7fd201938640 10 monclient: monmap: 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 librados: starting msgr at 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 librados: starting objecter 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 Processor -- start 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 -- start start 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 librados: setting wanted keys 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 librados: calling monclient init 2024-03-25T19:33:49.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient: init 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient(hunting): picked mon.noname-a con 0x7fd1fc0a9920 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient(hunting): picked mon.noname-b con 0x7fd1fc13fce0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient(hunting): picked mon.noname-c con 0x7fd1fc142ee0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc1466d0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0aba10 con 0x7fd1fc13fce0 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0abcd0 con 0x7fd1fc142ee0 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.807+0000 7fd201938640 10 monclient(hunting): authenticate will time out at 6117.095473s 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1faffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1fc0a9920 0x7fd1fc145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.196:40984/0 (socket says 172.21.15.196:40984) 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1faffd640 1 -- 172.21.15.196:0/455736794 learned_addr learned my addr 172.21.15.196:0/455736794 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1fc0a9920 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1fc142ee0 2024-03-25T19:33:49.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.808+0000 7fd1f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167597442 0 0) 0x7fd1fc1466d0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.810+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1dc004eb0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105381670 0 0) 0x7fd1fc0abcd0 con 0x7fd1fc142ee0 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.811+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1fc1466d0 con 0x7fd1fc142ee0 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.814+0000 7fd1f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1fc13fce0 2024-03-25T19:33:49.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.814+0000 7fd1f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.815 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.815+0000 7f9936441040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:33:49.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137705841 0 0) 0x7fd1dc004eb0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1fc0abcd0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd1ec001a10 con 0x7fd1fc0a9920 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.816 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.815+0000 7fd1f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591596151 0 0) 0x7fd1fc0aba10 con 0x7fd1fc13fce0 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:33:49.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.816+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1dc004eb0 con 0x7fd1fc13fce0 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3350635633 0 0) 0x7fd1fc0abcd0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 >> v1:172.21.15.29:6790/0 conn(0x7fd1fc142ee0 legacy=0x7fd1fc1e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 >> v1:172.21.15.196:6789/0 conn(0x7fd1fc13fce0 legacy=0x7fd1fc1da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 1 monclient: found mon.a 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:49.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0abe60 con 0x7fd1fc0a9920 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd1f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 10 monclient: _renew_subs 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1fc146280 con 0x7fd1fc0a9920 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 10 monclient: _renew_subs 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1fc1db130 con 0x7fd1fc0a9920 2024-03-25T19:33:49.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.818+0000 7fd201938640 1 librados: init done 2024-03-25T19:33:49.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.819+0000 7fd201938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.819+0000 7fd201938640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.819+0000 7fd201938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.819+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1fc1db3e0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.869 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.870+0000 7f9936441040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:33:49.878 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.879+0000 7f3ab5e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:33:49.916 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:49.916+0000 7f9936441040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:33:49.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1ec001c70 con 0x7fd1fc0a9920 2024-03-25T19:33:49.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:49.921 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd1ec003a60 con 0x7fd1fc0a9920 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 10 monclient: dump: 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:49.922 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:49.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f9ffb640 4 set_mon_vals no callback set 2024-03-25T19:33:49.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd1ec004010 con 0x7fd1fc0a9920 2024-03-25T19:33:49.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1037100310 0 0) 0x7fd1ec006680 con 0x7fd1fc0a9920 2024-03-25T19:33:49.923 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 10 client.4125.objecter ms_dispatch 0x7fd1fc000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.922+0000 7fd1f8ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.923+0000 7fd1f8ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.923+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd1ec0069d0 con 0x7fd1fc0a9920 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.923+0000 7fd1f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:49.924 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.923+0000 7fd1f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:49.927 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.928+0000 7f3ab5e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:33:49.982 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:49.983+0000 7f3ab5e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:33:49.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.996+0000 7fd201938640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "91604cd3-71e6-4c46-ab34-7b0d787ad733", "id": 1}] 2024-03-25T19:33:49.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.996+0000 7fd201938640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "91604cd3-71e6-4c46-ab34-7b0d787ad733", "id": 1}] 2024-03-25T19:33:49.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.996+0000 7fd201938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:49.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:49.996+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd new", "uuid": "91604cd3-71e6-4c46-ab34-7b0d787ad733", "id": 1} v 0) v1 -- 0x7fd1fc04f730 con 0x7fd1fc0a9920 2024-03-25T19:33:50.033 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.034+0000 7f3ab5e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:33:50.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd1f8ff9640 1 -- 172.21.15.196:0/455736794 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "91604cd3-71e6-4c46-ab34-7b0d787ad733", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2783094425 0 940483276) 0x7fd1ec0057c0 con 0x7fd1fc0a9920 2024-03-25T19:33:50.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd1f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "91604cd3-71e6-4c46-ab34-7b0d787ad733", "id": 1}] 2024-03-25T19:33:50.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd1f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:50.036 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd201938640 10 librados: watch_flush enter 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd201938640 10 librados: watch_flush exit 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd201938640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.036+0000 7fd201938640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 10 monclient: shutdown 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 >> v1:172.21.15.29:6789/0 conn(0x7fd1fc0a9920 legacy=0x7fd1fc145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 shutdown_connections 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 >> 172.21.15.196:0/455736794 conn(0x7fd1fc0b7df0 msgr2=0x7fd1fc0b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 shutdown_connections 2024-03-25T19:33:50.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 -- 172.21.15.196:0/455736794 wait complete. 2024-03-25T19:33:50.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.037+0000 7fd201938640 1 librados: shutdown 2024-03-25T19:33:50.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f 2 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.152+0000 7fdf2535d640 10 monclient: get_monmap_and_config 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.152+0000 7fdf2535d640 10 monclient: build_initial_monmap 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.152+0000 7fdf2535d640 10 monclient: monmap: 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:50.153465+0000 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:50.153465+0000 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.152 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:50.153 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:50.153 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:50.153 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 1 Processor -- start 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 1 -- start start 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient: init 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:50.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf200a9fa0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf200ab520 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf200afc70 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200ac500 con 0x7fdf200a9fa0 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200ab040 con 0x7fdf200afc70 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf20072a30 con 0x7fdf200ab520 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.153+0000 7fdf2535d640 10 monclient(hunting): authenticate will time out at 6117.441297s 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.154+0000 7fdf1f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf200afc70 0x7fdf2014a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51282/0 (socket says 172.21.15.196:51282) 2024-03-25T19:33:50.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.154+0000 7fdf1f7fe640 1 -- 172.21.15.196:0/1728804807 learned_addr learned my addr 172.21.15.196:0/1728804807 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501214461 0 0) 0x7fdf200ac500 con 0x7fdf200a9fa0 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:33:50.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf08003520 con 0x7fdf200a9fa0 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210772001 0 0) 0x7fdf200ab040 con 0x7fdf200afc70 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf200ac500 con 0x7fdf200afc70 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025691379 0 0) 0x7fdf20072a30 con 0x7fdf200ab520 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:33:50.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.156+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf200ab040 con 0x7fdf200ab520 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323733369 0 0) 0x7fdf200ac500 con 0x7fdf200afc70 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 >> v1:172.21.15.29:6790/0 conn(0x7fdf200ab520 legacy=0x7fdf2013bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 >> v1:172.21.15.29:6789/0 conn(0x7fdf200a9fa0 legacy=0x7fdf200af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 1 monclient: found mon.noname-c 2024-03-25T19:33:50.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf200ac770 con 0x7fdf200afc70 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf2535d640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:33:50.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.159+0000 7fdf2535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.162+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf10001630 con 0x7fdf200afc70 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.162+0000 7fdf1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.162+0000 7fdf1dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.162+0000 7fdf1dffb640 10 monclient: dump: 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.162+0000 7fdf2535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:50.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf10001810 con 0x7fdf200afc70 2024-03-25T19:33:50.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 1 -- 172.21.15.196:0/1728804807 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf10001ba0 con 0x7fdf200afc70 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf1dffb640 10 monclient: dump: 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.186 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.187 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.187 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf2535d640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:50.187 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.186+0000 7fdf2535d640 4 set_mon_vals no callback set 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 10 monclient: shutdown 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 1 -- 172.21.15.196:0/1728804807 >> v1:172.21.15.196:6789/0 conn(0x7fdf200afc70 legacy=0x7fdf2014a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 1 -- 172.21.15.196:0/1728804807 shutdown_connections 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 1 -- 172.21.15.196:0/1728804807 >> 172.21.15.196:0/1728804807 conn(0x7fdf2006b560 msgr2=0x7fdf200b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 1 -- 172.21.15.196:0/1728804807 shutdown_connections 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.187+0000 7fdf2535d640 1 -- 172.21.15.196:0/1728804807 wait complete. 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 10 monclient: build_initial_monmap 2024-03-25T19:33:50.189 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 10 monclient: monmap: 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 librados: starting msgr at 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 librados: starting objecter 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 Processor -- start 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 -- start start 2024-03-25T19:33:50.190 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 librados: setting wanted keys 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 1 librados: calling monclient init 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.188+0000 7fdf2535d640 10 monclient: init 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf200a9fa0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf200afc70 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf20132440 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf201c90d0 con 0x7fdf200afc70 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf201d7b50 con 0x7fdf20132440 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf201d77e0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf2535d640 10 monclient(hunting): authenticate will time out at 6117.476852s 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf1e7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdf200afc70 0x7fdf20131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.196:41012/0 (socket says 172.21.15.196:41012) 2024-03-25T19:33:50.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.189+0000 7fdf1e7fc640 1 -- 172.21.15.196:0/689122634 learned_addr learned my addr 172.21.15.196:0/689122634 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.190+0000 7fdf077fe640 10 client.?.objecter ms_handle_connect 0x7fdf200afc70 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.190+0000 7fdf077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.190+0000 7fdf077fe640 10 client.?.objecter ms_handle_connect 0x7fdf200a9fa0 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.190+0000 7fdf077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.191+0000 7fdf077fe640 10 client.?.objecter ms_handle_connect 0x7fdf20132440 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.191+0000 7fdf077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939634820 0 0) 0x7fdf201d77e0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c004eb0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124298817 0 0) 0x7fdf201c90d0 con 0x7fdf200afc70 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.192+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf201d77e0 con 0x7fdf200afc70 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831647202 0 0) 0x7fdf201d7b50 con 0x7fdf20132440 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:33:50.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.193+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf201c90d0 con 0x7fdf20132440 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402099922 0 0) 0x7fdf201d77e0 con 0x7fdf200afc70 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf201d7b50 con 0x7fdf200afc70 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf140019f0 con 0x7fdf200afc70 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:50.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 10 monclient(hunting): dump: 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.196+0000 7fdf077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2315847186 0 0) 0x7fdf0c004eb0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf201d77e0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf18001cd0 con 0x7fdf200a9fa0 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.197+0000 7fdf077fe640 10 monclient(hunting): dump: 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.198 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560542503 0 0) 0x7fdf201c90d0 con 0x7fdf20132440 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0c004eb0 con 0x7fdf20132440 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf10001b20 con 0x7fdf20132440 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.198+0000 7fdf077fe640 10 monclient(hunting): dump: 2024-03-25T19:33:50.199 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.200 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3173372586 0 0) 0x7fdf201d7b50 con 0x7fdf200afc70 2024-03-25T19:33:50.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 >> v1:172.21.15.29:6790/0 conn(0x7fdf200a9fa0 legacy=0x7fdf201d8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 >> v1:172.21.15.196:6789/0 conn(0x7fdf20132440 legacy=0x7fdf201c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 1 monclient: found mon.a 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2013c890 con 0x7fdf200afc70 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 10 monclient: _renew_subs 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2013cae0 con 0x7fdf200afc70 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 10 monclient: _renew_subs 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf201ca3d0 con 0x7fdf200afc70 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.202+0000 7fdf2535d640 1 librados: init done 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.203+0000 7fdf2535d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.203+0000 7fdf2535d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.203+0000 7fdf2535d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:50.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.203+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf201d7ce0 con 0x7fdf200afc70 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf14003610 con 0x7fdf200afc70 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdf140039a0 con 0x7fdf200afc70 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 10 monclient: dump: 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.207 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdf14003f50 con 0x7fdf200afc70 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf1d7fa640 4 set_mon_vals no callback set 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3733178632 0 0) 0x7fdf140066b0 con 0x7fdf200afc70 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 10 client.4131.objecter ms_dispatch 0x7fdf20000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:33:50.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.207+0000 7fdf077fe640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:33:50.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.307+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdf140063a0 con 0x7fdf200afc70 2024-03-25T19:33:50.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.307+0000 7fdf077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.307 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.307+0000 7fdf077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:50.317 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:50.317+0000 7f9936441040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:33:50.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.379+0000 7fdf2535d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f", "id": 2}] 2024-03-25T19:33:50.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.379+0000 7fdf2535d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f", "id": 2}] 2024-03-25T19:33:50.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.379+0000 7fdf2535d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:50.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.379+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f", "id": 2} v 0) v1 -- 0x7fdf201d7970 con 0x7fdf200afc70 2024-03-25T19:33:50.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.428+0000 7fdf077fe640 1 -- 172.21.15.196:0/689122634 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2140980757 0 216599125) 0x7fdf140042d0 con 0x7fdf200afc70 2024-03-25T19:33:50.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f", "id": 2}] 2024-03-25T19:33:50.428 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:50.429 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 10 librados: watch_flush enter 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 10 librados: watch_flush exit 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 10 monclient: shutdown 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.429+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 >> v1:172.21.15.29:6789/0 conn(0x7fdf200afc70 legacy=0x7fdf20131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.430+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 shutdown_connections 2024-03-25T19:33:50.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.430+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 >> 172.21.15.196:0/689122634 conn(0x7fdf2006b560 msgr2=0x7fdf200b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.430+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 shutdown_connections 2024-03-25T19:33:50.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.430+0000 7fdf2535d640 1 -- 172.21.15.196:0/689122634 wait complete. 2024-03-25T19:33:50.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.430+0000 7fdf2535d640 1 librados: shutdown 2024-03-25T19:33:50.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 8396ed08-a86d-47b2-80d4-31af705ae951 3 2024-03-25T19:33:50.447 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.448+0000 7f3ab5e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:33:50.455 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:50.456+0000 7f9936441040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.540+0000 7fb496954640 10 monclient: get_monmap_and_config 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.540+0000 7fb496954640 10 monclient: build_initial_monmap 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.540+0000 7fb496954640 10 monclient: monmap: 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.540 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:50.541222+0000 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:50.541222+0000 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.540+0000 7fb496954640 1 Processor -- start 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 1 -- start start 2024-03-25T19:33:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient: init 2024-03-25T19:33:50.542 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient(hunting): picked mon.noname-a con 0x7fb4900b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient(hunting): picked mon.noname-b con 0x7fb490142a20 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient(hunting): picked mon.noname-c con 0x7fb4900a96b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900a9d60 con 0x7fb4900b4700 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49013ff10 con 0x7fb4900a96b0 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900aa1f0 con 0x7fb490142a20 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.541+0000 7fb496954640 10 monclient(hunting): authenticate will time out at 6117.829170s 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.542+0000 7fb494ef1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb4900a96b0 0x7fb49013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51314/0 (socket says 172.21.15.196:51314) 2024-03-25T19:33:50.543 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.542+0000 7fb494ef1640 1 -- 172.21.15.196:0/2358378096 learned_addr learned my addr 172.21.15.196:0/2358378096 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441861031 0 0) 0x7fb49013ff10 con 0x7fb4900a96b0 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:33:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.544+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb478003520 con 0x7fb4900a96b0 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309336737 0 0) 0x7fb4900aa1f0 con 0x7fb490142a20 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49013ff10 con 0x7fb490142a20 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184073833 0 0) 0x7fb4900a9d60 con 0x7fb4900b4700 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:33:50.545 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.545+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4900aa1f0 con 0x7fb4900b4700 2024-03-25T19:33:50.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398579161 0 0) 0x7fb478003520 con 0x7fb4900a96b0 2024-03-25T19:33:50.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 >> v1:172.21.15.29:6790/0 conn(0x7fb490142a20 legacy=0x7fb49013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 >> v1:172.21.15.29:6789/0 conn(0x7fb4900b4700 legacy=0x7fb490142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.547 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 monclient: found mon.noname-c 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900b4ae0 con 0x7fb4900a96b0 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: _finish_auth 0 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb488001a10 con 0x7fb4900a96b0 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb48effd640 10 monclient: dump: 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.548 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb496954640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:33:50.549 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.547+0000 7fb496954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:50.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb488001b70 con 0x7fb4900a96b0 2024-03-25T19:33:50.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:50.550 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 1 -- 172.21.15.196:0/2358378096 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb488001f80 con 0x7fb4900a96b0 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb48effd640 10 monclient: dump: 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:50.551 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 4 set_mon_vals no callback set 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 10 monclient: shutdown 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 1 -- 172.21.15.196:0/2358378096 >> v1:172.21.15.196:6789/0 conn(0x7fb4900a96b0 legacy=0x7fb49013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 1 -- 172.21.15.196:0/2358378096 shutdown_connections 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 1 -- 172.21.15.196:0/2358378096 >> 172.21.15.196:0/2358378096 conn(0x7fb4900668e0 msgr2=0x7fb490066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 1 -- 172.21.15.196:0/2358378096 shutdown_connections 2024-03-25T19:33:50.552 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.550+0000 7fb496954640 1 -- 172.21.15.196:0/2358378096 wait complete. 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.551+0000 7fb496954640 10 monclient: build_initial_monmap 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.551+0000 7fb496954640 10 monclient: monmap: 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:50.554 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 librados: starting msgr at 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 librados: starting objecter 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 Processor -- start 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 -- start start 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 librados: setting wanted keys 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 librados: calling monclient init 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient: init 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient(hunting): picked mon.noname-a con 0x7fb4900a96b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:50.555 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient(hunting): picked mon.noname-b con 0x7fb4900b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient(hunting): picked mon.noname-c con 0x7fb490142a20 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900af020 con 0x7fb4900a96b0 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900ae210 con 0x7fb4900b4700 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900ae4f0 con 0x7fb490142a20 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.552+0000 7fb496954640 10 monclient(hunting): authenticate will time out at 6117.840541s 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb4900b4700 0x7fb4900acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51316/0 (socket says 172.21.15.196:51316) 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48f7fe640 1 -- 172.21.15.196:0/1758731217 learned_addr learned my addr 172.21.15.196:0/1758731217 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter ms_handle_connect 0x7fb4900b4700 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter ms_handle_connect 0x7fb4900a96b0 2024-03-25T19:33:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter ms_handle_connect 0x7fb490142a20 2024-03-25T19:33:50.557 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.553+0000 7fb48cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909522564 0 0) 0x7fb4900ae210 con 0x7fb4900b4700 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47c004eb0 con 0x7fb4900b4700 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950172637 0 0) 0x7fb4900af020 con 0x7fb4900a96b0 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4900ae210 con 0x7fb4900a96b0 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309903592 0 0) 0x7fb4900ae4f0 con 0x7fb490142a20 2024-03-25T19:33:50.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:50.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:50.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:50.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:33:50.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.555+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4900af020 con 0x7fb490142a20 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233619599 0 0) 0x7fb47c004eb0 con 0x7fb4900b4700 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4900ae4f0 con 0x7fb4900b4700 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb480001a30 con 0x7fb4900b4700 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:50.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): dump: 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1195991740 0 0) 0x7fb4900ae210 con 0x7fb4900a96b0 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb47c004eb0 con 0x7fb4900a96b0 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb484001870 con 0x7fb4900a96b0 2024-03-25T19:33:50.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): dump: 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1837778543 0 0) 0x7fb4900af020 con 0x7fb490142a20 2024-03-25T19:33:50.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4900ae210 con 0x7fb490142a20 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb488001e80 con 0x7fb490142a20 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.560+0000 7fb48cff9640 10 monclient(hunting): dump: 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.564 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.565 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.565 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3515227168 0 0) 0x7fb4900ae210 con 0x7fb490142a20 2024-03-25T19:33:50.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 >> v1:172.21.15.196:6789/0 conn(0x7fb4900b4700 legacy=0x7fb4900acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 >> v1:172.21.15.29:6789/0 conn(0x7fb4900a96b0 legacy=0x7fb4900ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 1 monclient: found mon.c 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900ac0e0 con 0x7fb490142a20 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb48cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 5 monclient: authenticate success, global_id 4130 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 10 monclient: _renew_subs 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4900ad340 con 0x7fb490142a20 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 10 monclient: _renew_subs 2024-03-25T19:33:50.567 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49013b070 con 0x7fb490142a20 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.563+0000 7fb496954640 1 librados: init done 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.564+0000 7fb496954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.564+0000 7fb496954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.564+0000 7fb496954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:33:50.568 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.564+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49004f730 con 0x7fb490142a20 2024-03-25T19:33:50.592 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.593+0000 7f3ab5e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:33:50.717 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.718+0000 7f3ab5e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:33:50.765 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.766+0000 7f3ab5e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb488001560 con 0x7fb490142a20 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb488004e10 con 0x7fb490142a20 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 10 monclient: dump: 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:50.792 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:50.793 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:50.793 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:50.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb488005300 con 0x7fb490142a20 2024-03-25T19:33:50.793 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.792+0000 7fb48e7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:50.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2254641748 0 0) 0x7fb4880068a0 con 0x7fb490142a20 2024-03-25T19:33:50.794 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 10 client.4130.objecter ms_dispatch 0x7fb490000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 3 client.4130.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 3 client.4130.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb488006bb0 con 0x7fb490142a20 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.795 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.795+0000 7fb48cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:50.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.864+0000 7fb496954640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8396ed08-a86d-47b2-80d4-31af705ae951", "id": 3}] 2024-03-25T19:33:50.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.864+0000 7fb496954640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8396ed08-a86d-47b2-80d4-31af705ae951", "id": 3}] 2024-03-25T19:33:50.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.864+0000 7fb496954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:33:50.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.864+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd new", "uuid": "8396ed08-a86d-47b2-80d4-31af705ae951", "id": 3} v 0) v1 -- 0x7fb49013b320 con 0x7fb490142a20 2024-03-25T19:33:50.883 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.884+0000 7f3ab5e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:33:50.889 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:50.889+0000 7f9936441040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:33:50.934 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:50.934+0000 7f9936441040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:33:50.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.980+0000 7fb48cff9640 1 -- 172.21.15.196:0/1758731217 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8396ed08-a86d-47b2-80d4-31af705ae951", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 570713335 0 525046306) 0x7fb4880064e0 con 0x7fb490142a20 2024-03-25T19:33:50.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.980+0000 7fb48cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8396ed08-a86d-47b2-80d4-31af705ae951", "id": 3}] 2024-03-25T19:33:50.980 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.980+0000 7fb48cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:50.981 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 10 librados: watch_flush enter 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 10 librados: watch_flush exit 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 10 client.4130.objecter successfully canceled tick 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 10 monclient: shutdown 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 >> v1:172.21.15.29:6790/0 conn(0x7fb490142a20 legacy=0x7fb4901e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.982 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 shutdown_connections 2024-03-25T19:33:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 >> 172.21.15.196:0/1758731217 conn(0x7fb4900668e0 msgr2=0x7fb4900b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 shutdown_connections 2024-03-25T19:33:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.981+0000 7fb496954640 1 -- 172.21.15.196:0/1758731217 wait complete. 2024-03-25T19:33:50.983 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:50.982+0000 7fb496954640 1 librados: shutdown 2024-03-25T19:33:50.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new ca31575f-d609-411c-97df-72ed610cc543 4 2024-03-25T19:33:50.995 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:50.996+0000 7f3ab5e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:33:51.052 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:51.052+0000 7f9936441040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 10 monclient: get_monmap_and_config 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 10 monclient: build_initial_monmap 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 10 monclient: monmap: 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:51.133260+0000 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:51.133260+0000 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:51.133 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 1 Processor -- start 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 1 -- start start 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.132+0000 7fe03556b640 10 monclient: init 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0300b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-a con 0x7fe030142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-b con 0x7fe0300a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:51.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0300a9d60 con 0x7fe030142a20 2024-03-25T19:33:51.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03013ff10 con 0x7fe0300b4700 2024-03-25T19:33:51.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0300aa1f0 con 0x7fe0300a96b0 2024-03-25T19:33:51.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:51.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.133+0000 7fe03556b640 10 monclient(hunting): authenticate will time out at 6118.421145s 2024-03-25T19:33:51.136 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.137+0000 7fe02effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe0300b4700 0x7fe030142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51332/0 (socket says 172.21.15.196:51332) 2024-03-25T19:33:51.137 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.137+0000 7fe02effd640 1 -- 172.21.15.196:0/2144232444 learned_addr learned my addr 172.21.15.196:0/2144232444 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:51.140 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068387157 0 0) 0x7fe03013ff10 con 0x7fe0300b4700 2024-03-25T19:33:51.141 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.141 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.141 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.141 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:33:51.141 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.141+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018003520 con 0x7fe0300b4700 2024-03-25T19:33:51.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763662099 0 0) 0x7fe018003520 con 0x7fe0300b4700 2024-03-25T19:33:51.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 >> v1:172.21.15.29:6790/0 conn(0x7fe0300a96b0 legacy=0x7fe03013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:51.146 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 >> v1:172.21.15.29:6789/0 conn(0x7fe030142a20 legacy=0x7fe03013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 monclient: found mon.noname-c 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0300b4ae0 con 0x7fe0300b4700 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe020001690 con 0x7fe0300b4700 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe03556b640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe03556b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:51.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe02dffb640 10 monclient: dump: 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.146+0000 7fe03556b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:51.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe020001860 con 0x7fe0300b4700 2024-03-25T19:33:51.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:51.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 1 -- 172.21.15.196:0/2144232444 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe020001cb0 con 0x7fe0300b4700 2024-03-25T19:33:51.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe02dffb640 10 monclient: dump: 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe03556b640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:51.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.147+0000 7fe03556b640 4 set_mon_vals no callback set 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 10 monclient: shutdown 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 1 -- 172.21.15.196:0/2144232444 >> v1:172.21.15.196:6789/0 conn(0x7fe0300b4700 legacy=0x7fe030142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 1 -- 172.21.15.196:0/2144232444 shutdown_connections 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 1 -- 172.21.15.196:0/2144232444 >> 172.21.15.196:0/2144232444 conn(0x7fe0300668e0 msgr2=0x7fe030066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 1 -- 172.21.15.196:0/2144232444 shutdown_connections 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.148+0000 7fe03556b640 1 -- 172.21.15.196:0/2144232444 wait complete. 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.149+0000 7fe03556b640 10 monclient: build_initial_monmap 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.149+0000 7fe03556b640 10 monclient: monmap: 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:51.153 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.149+0000 7fe03556b640 1 librados: starting msgr at 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.149+0000 7fe03556b640 1 librados: starting objecter 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 Processor -- start 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 -- start start 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 librados: setting wanted keys 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 librados: calling monclient init 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient: init 2024-03-25T19:33:51.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0300a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0300b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient(hunting): picked mon.noname-b con 0x7fe030142a20 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0301d75c0 con 0x7fe0300b4700 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03013a400 con 0x7fe030142a20 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe030137da0 con 0x7fe0300a96b0 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.150+0000 7fe03556b640 10 monclient(hunting): authenticate will time out at 6118.438294s 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.151+0000 7fe02f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe030142a20 0x7fe030130f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51344/0 (socket says 172.21.15.196:51344) 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.151+0000 7fe02f7fe640 1 -- 172.21.15.196:0/806559881 learned_addr learned my addr 172.21.15.196:0/806559881 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:51.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.151+0000 7fe0177fe640 10 client.?.objecter ms_handle_connect 0x7fe030142a20 2024-03-25T19:33:51.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.151+0000 7fe0177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.152+0000 7fe0177fe640 10 client.?.objecter ms_handle_connect 0x7fe0300a96b0 2024-03-25T19:33:51.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.152+0000 7fe0177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941842372 0 0) 0x7fe03013a400 con 0x7fe030142a20 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.153+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c004eb0 con 0x7fe030142a20 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.154+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149881489 0 0) 0x7fe030137da0 con 0x7fe0300a96b0 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.154+0000 7fe0177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.154+0000 7fe0177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.154+0000 7fe0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.155+0000 7fe0177fe640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:33:51.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.155+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe03013a400 con 0x7fe0300a96b0 2024-03-25T19:33:51.158 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:51.155+0000 7f9936441040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:33:51.158 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.156+0000 7fe0177fe640 10 client.?.objecter ms_handle_connect 0x7fe0300b4700 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.156+0000 7fe0177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.157+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756273584 0 0) 0x7fe01c004eb0 con 0x7fe030142a20 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.157+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe030137da0 con 0x7fe030142a20 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.158+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe028001b90 con 0x7fe030142a20 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.158+0000 7fe0177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.158+0000 7fe0177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.158+0000 7fe0177fe640 10 monclient(hunting): dump: 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.159 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.160 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.160 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.160 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.158+0000 7fe0177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768535971 0 0) 0x7fe0301d75c0 con 0x7fe0300b4700 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.160+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c004eb0 con 0x7fe0300b4700 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143627598 0 0) 0x7fe03013a400 con 0x7fe0300a96b0 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0301d75c0 con 0x7fe0300a96b0 2024-03-25T19:33:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe020001b70 con 0x7fe0300a96b0 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient(hunting): dump: 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.163 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2338774828 0 0) 0x7fe030137da0 con 0x7fe030142a20 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 >> v1:172.21.15.29:6790/0 conn(0x7fe0300a96b0 legacy=0x7fe0301d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 >> v1:172.21.15.29:6789/0 conn(0x7fe0300b4700 legacy=0x7fe0301351e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 monclient: found mon.b 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0301391e0 con 0x7fe030142a20 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe0177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 10 monclient: _renew_subs 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe030138080 con 0x7fe030142a20 2024-03-25T19:33:51.164 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 10 monclient: _renew_subs 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe030138d00 con 0x7fe030142a20 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.161+0000 7fe03556b640 1 librados: init done 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.162+0000 7fe03556b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.162+0000 7fe03556b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.162+0000 7fe03556b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.165 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.162+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03004f730 con 0x7fe030142a20 2024-03-25T19:33:51.167 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:51.168+0000 7f3ab5e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:33:51.212 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:51.213+0000 7f3ab5e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:33:51.316 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:51.317+0000 7f9936441040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe028002020 con 0x7fe030142a20 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe028004d80 con 0x7fe030142a20 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 10 monclient: dump: 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.357 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe028005270 con 0x7fe030142a20 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe02d7fa640 4 set_mon_vals no callback set 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 853502422 0 0) 0x7fe028006a00 con 0x7fe030142a20 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.356+0000 7fe0177fe640 10 client.4135.objecter ms_dispatch 0x7fe030000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:33:51.358 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:33:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:33:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T19:33:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe028018e00 con 0x7fe030142a20 2024-03-25T19:33:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.357+0000 7fe0177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:51.361 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:51.361+0000 7f9936441040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:33:51.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:51.373+0000 7f3ab5e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:33:51.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.429+0000 7fe03556b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca31575f-d609-411c-97df-72ed610cc543", "id": 4}] 2024-03-25T19:33:51.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.429+0000 7fe03556b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca31575f-d609-411c-97df-72ed610cc543", "id": 4}] 2024-03-25T19:33:51.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.429+0000 7fe03556b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.429 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.429+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ca31575f-d609-411c-97df-72ed610cc543", "id": 4} v 0) v1 -- 0x7fe030069f30 con 0x7fe030142a20 2024-03-25T19:33:51.520 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:51.521+0000 7f9936441040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:33:51.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.525+0000 7fe0177fe640 1 -- 172.21.15.196:0/806559881 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca31575f-d609-411c-97df-72ed610cc543", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 800715269 0 1697012583) 0x7fe028006500 con 0x7fe030142a20 2024-03-25T19:33:51.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.525+0000 7fe0177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca31575f-d609-411c-97df-72ed610cc543", "id": 4}] 2024-03-25T19:33:51.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.525+0000 7fe0177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:51.525 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 10 librados: watch_flush enter 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 10 librados: watch_flush exit 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 10 client.4135.objecter successfully canceled tick 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 10 monclient: shutdown 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 >> v1:172.21.15.196:6789/0 conn(0x7fe030142a20 legacy=0x7fe030130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 shutdown_connections 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 >> 172.21.15.196:0/806559881 conn(0x7fe0300668e0 msgr2=0x7fe03013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.526+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 shutdown_connections 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.527+0000 7fe03556b640 1 -- 172.21.15.196:0/806559881 wait complete. 2024-03-25T19:33:51.527 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.527+0000 7fe03556b640 1 librados: shutdown 2024-03-25T19:33:51.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 7a6f10ae-99ec-40f1-96a6-9467bde53124 5 2024-03-25T19:33:51.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 10 monclient: get_monmap_and_config 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 10 monclient: build_initial_monmap 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 10 monclient: monmap: 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:51.679021+0000 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:51.679021+0000 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:51.678 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 1 Processor -- start 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 1 -- start start 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.678+0000 7fd2df14d640 10 monclient: init 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d80b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d8142a20 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d80a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80a9d60 con 0x7fd2d80b4700 2024-03-25T19:33:51.680 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d813ff10 con 0x7fd2d8142a20 2024-03-25T19:33:51.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80aa1f0 con 0x7fd2d80a96b0 2024-03-25T19:33:51.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:51.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.679+0000 7fd2df14d640 10 monclient(hunting): authenticate will time out at 6118.967047s 2024-03-25T19:33:51.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.680+0000 7fd2d7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd2d8142a20 0x7fd2d813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51358/0 (socket says 172.21.15.196:51358) 2024-03-25T19:33:51.681 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.680+0000 7fd2d7fff640 1 -- 172.21.15.196:0/4086959006 learned_addr learned my addr 172.21.15.196:0/4086959006 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452820157 0 0) 0x7fd2d80aa1f0 con 0x7fd2d80a96b0 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c8003520 con 0x7fd2d80a96b0 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128787037 0 0) 0x7fd2d813ff10 con 0x7fd2d8142a20 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:33:51.682 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.682+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d80aa1f0 con 0x7fd2d8142a20 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226311724 0 0) 0x7fd2d80a9d60 con 0x7fd2d80b4700 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:33:51.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.683+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d813ff10 con 0x7fd2d80b4700 2024-03-25T19:33:51.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3599312598 0 0) 0x7fd2d80aa1f0 con 0x7fd2d8142a20 2024-03-25T19:33:51.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 >> v1:172.21.15.29:6790/0 conn(0x7fd2d80a96b0 legacy=0x7fd2d813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 >> v1:172.21.15.29:6789/0 conn(0x7fd2d80b4700 legacy=0x7fd2d8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80b4ae0 con 0x7fd2d8142a20 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2c4001670 con 0x7fd2d8142a20 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2d77fe640 10 monclient: dump: 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2df14d640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:33:51.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.685+0000 7fd2df14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:51.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c40017d0 con 0x7fd2d8142a20 2024-03-25T19:33:51.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:51.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:51.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 4 set_mon_vals no callback set 2024-03-25T19:33:51.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 1 -- 172.21.15.196:0/4086959006 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2c4001be0 con 0x7fd2d8142a20 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2d77fe640 10 monclient: dump: 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 10 monclient: shutdown 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 1 -- 172.21.15.196:0/4086959006 >> v1:172.21.15.196:6789/0 conn(0x7fd2d8142a20 legacy=0x7fd2d813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.687+0000 7fd2df14d640 1 -- 172.21.15.196:0/4086959006 shutdown_connections 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.688+0000 7fd2df14d640 1 -- 172.21.15.196:0/4086959006 >> 172.21.15.196:0/4086959006 conn(0x7fd2d80668e0 msgr2=0x7fd2d8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.688+0000 7fd2df14d640 1 -- 172.21.15.196:0/4086959006 shutdown_connections 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.688+0000 7fd2df14d640 1 -- 172.21.15.196:0/4086959006 wait complete. 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.688+0000 7fd2df14d640 10 monclient: build_initial_monmap 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.688+0000 7fd2df14d640 10 monclient: monmap: 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:51.692 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 librados: starting msgr at 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 librados: starting objecter 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 Processor -- start 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 -- start start 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 librados: setting wanted keys 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 librados: calling monclient init 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient: init 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:51.693 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d80a96b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d80b4700 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d8142a20 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d813c880 con 0x7fd2d80a96b0 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80ac300 con 0x7fd2d80b4700 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d813c510 con 0x7fd2d8142a20 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.689+0000 7fd2df14d640 10 monclient(hunting): authenticate will time out at 6118.977500s 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd2d80b4700 0x7fd2d80ac790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51374/0 (socket says 172.21.15.196:51374) 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d7fff640 1 -- 172.21.15.196:0/2657050339 learned_addr learned my addr 172.21.15.196:0/2657050339 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter ms_handle_connect 0x7fd2d80b4700 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter ms_handle_connect 0x7fd2d80a96b0 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter ms_handle_connect 0x7fd2d8142a20 2024-03-25T19:33:51.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.690+0000 7fd2d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047659765 0 0) 0x7fd2d80ac300 con 0x7fd2d80b4700 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc004eb0 con 0x7fd2d80b4700 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985461036 0 0) 0x7fd2d813c510 con 0x7fd2d8142a20 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.692+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d80ac300 con 0x7fd2d8142a20 2024-03-25T19:33:51.696 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988713947 0 0) 0x7fd2d813c880 con 0x7fd2d80a96b0 2024-03-25T19:33:51.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:51.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:51.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:51.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:33:51.697 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.693+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d813c510 con 0x7fd2d80a96b0 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165502108 0 0) 0x7fd2bc004eb0 con 0x7fd2d80b4700 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d813c880 con 0x7fd2d80b4700 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2c4001670 con 0x7fd2d80b4700 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:51.698 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): dump: 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184690502 0 0) 0x7fd2d80ac300 con 0x7fd2d8142a20 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2bc004eb0 con 0x7fd2d8142a20 2024-03-25T19:33:51.699 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2d0001c50 con 0x7fd2d8142a20 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.697+0000 7fd2d57fa640 10 monclient(hunting): dump: 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.700 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4005325998 0 0) 0x7fd2d813c510 con 0x7fd2d80a96b0 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d80ac300 con 0x7fd2d80a96b0 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2cc001830 con 0x7fd2d80a96b0 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.698+0000 7fd2d57fa640 10 monclient(hunting): dump: 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.701 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 641935278 0 0) 0x7fd2d813c880 con 0x7fd2d80b4700 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 >> v1:172.21.15.29:6790/0 conn(0x7fd2d8142a20 legacy=0x7fd2d81e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 >> v1:172.21.15.29:6789/0 conn(0x7fd2d80a96b0 legacy=0x7fd2d813be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 1 monclient: found mon.b 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80acea0 con 0x7fd2d80b4700 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:51.703 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 10 monclient: _renew_subs 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d813b260 con 0x7fd2d80b4700 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 10 monclient: _renew_subs 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d8139d70 con 0x7fd2d80b4700 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.700+0000 7fd2df14d640 1 librados: init done 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.701+0000 7fd2df14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.701+0000 7fd2df14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.701+0000 7fd2df14d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.704 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.701+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d804f730 con 0x7fd2d80b4700 2024-03-25T19:33:51.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c4004a90 con 0x7fd2d80b4700 2024-03-25T19:33:51.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:51.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd2c4004da0 con 0x7fd2d80b4700 2024-03-25T19:33:51.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:51.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 10 monclient: dump: 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd2c4005290 con 0x7fd2d80b4700 2024-03-25T19:33:51.706 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:33:51.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 633531448 0 0) 0x7fd2c40069a0 con 0x7fd2d80b4700 2024-03-25T19:33:51.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 10 client.4141.objecter ms_dispatch 0x7fd2d8000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:33:51.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:33:51.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:33:51.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.704+0000 7fd2d57fa640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T19:33:51.798 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.799+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd2c4006bb0 con 0x7fd2d80b4700 2024-03-25T19:33:51.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.799+0000 7fd2d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:51.799 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.799+0000 7fd2d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:51.870 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.870+0000 7fd2df14d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7a6f10ae-99ec-40f1-96a6-9467bde53124", "id": 5}] 2024-03-25T19:33:51.870 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.870+0000 7fd2df14d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7a6f10ae-99ec-40f1-96a6-9467bde53124", "id": 5}] 2024-03-25T19:33:51.870 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.870+0000 7fd2df14d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:51.870 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.870+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7a6f10ae-99ec-40f1-96a6-9467bde53124", "id": 5} v 0) v1 -- 0x7fd2d813fc90 con 0x7fd2d80b4700 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.937+0000 7fd2d57fa640 1 -- 172.21.15.196:0/2657050339 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a6f10ae-99ec-40f1-96a6-9467bde53124", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3905710478 0 1988412176) 0x7fd2c40064d0 con 0x7fd2d80b4700 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7a6f10ae-99ec-40f1-96a6-9467bde53124", "id": 5}] 2024-03-25T19:33:51.938 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:51.938 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-03-25T19:33:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 10 librados: watch_flush enter 2024-03-25T19:33:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 10 librados: watch_flush exit 2024-03-25T19:33:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T19:33:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 10 client.4141.objecter successfully canceled tick 2024-03-25T19:33:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 10 monclient: shutdown 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.938+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 >> v1:172.21.15.196:6789/0 conn(0x7fd2d80b4700 legacy=0x7fd2d80ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.939+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 shutdown_connections 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.939+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 >> 172.21.15.196:0/2657050339 conn(0x7fd2d80668e0 msgr2=0x7fd2d80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.939+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 shutdown_connections 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.939+0000 7fd2df14d640 1 -- 172.21.15.196:0/2657050339 wait complete. 2024-03-25T19:33:51.940 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:51.939+0000 7fd2df14d640 1 librados: shutdown 2024-03-25T19:33:51.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new cd5f7e4a-4d35-42f5-96a3-e362c57fd75a 6 2024-03-25T19:33:52.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.052+0000 7f42067f6640 10 monclient: get_monmap_and_config 2024-03-25T19:33:52.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.052+0000 7f42067f6640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.052+0000 7f42067f6640 10 monclient: monmap: 2024-03-25T19:33:52.052 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:52.052 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:52.053630+0000 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:52.053630+0000 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 1 Processor -- start 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 1 -- start start 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 10 monclient: init 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.053+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-b con 0x7f42000b4700 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4200142a20 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-c con 0x7f42000a96b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000a9d60 con 0x7f4200142a20 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420013ff10 con 0x7f42000a96b0 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000aa1f0 con 0x7f42000b4700 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f42067f6640 10 monclient(hunting): authenticate will time out at 6119.341648s 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f4204d93640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f42000a96b0 0x7f420013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51380/0 (socket says 172.21.15.196:51380) 2024-03-25T19:33:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.054+0000 7f4204d93640 1 -- 172.21.15.196:0/287539294 learned_addr learned my addr 172.21.15.196:0/287539294 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.056+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914338331 0 0) 0x7f420013ff10 con 0x7f42000a96b0 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.056+0000 7f41feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.056+0000 7f41feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:33:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003520 con 0x7f42000a96b0 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442726330 0 0) 0x7f42000aa1f0 con 0x7f42000b4700 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420013ff10 con 0x7f42000b4700 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498496711 0 0) 0x7f42000a9d60 con 0x7f4200142a20 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:33:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.057+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42000aa1f0 con 0x7f4200142a20 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803783687 0 0) 0x7f41e8003520 con 0x7f42000a96b0 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 >> v1:172.21.15.29:6790/0 conn(0x7f42000b4700 legacy=0x7f4200142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 >> v1:172.21.15.29:6789/0 conn(0x7f4200142a20 legacy=0x7f420013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 monclient: found mon.noname-c 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000b4ae0 con 0x7f42000a96b0 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f8001a10 con 0x7f42000a96b0 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f42067f6640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f42067f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f41feffd640 10 monclient: dump: 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.061 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.062 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.060+0000 7f42067f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f8001bf0 con 0x7f42000a96b0 2024-03-25T19:33:52.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f42067f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:52.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f42067f6640 4 set_mon_vals no callback set 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 1 -- 172.21.15.196:0/287539294 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f8002000 con 0x7f42000a96b0 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f41feffd640 10 monclient: dump: 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.067 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f42067f6640 10 monclient: shutdown 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f42067f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.061+0000 7f42067f6640 1 -- 172.21.15.196:0/287539294 >> v1:172.21.15.196:6789/0 conn(0x7f42000a96b0 legacy=0x7f420013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 -- 172.21.15.196:0/287539294 shutdown_connections 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 -- 172.21.15.196:0/287539294 >> 172.21.15.196:0/287539294 conn(0x7f42000668e0 msgr2=0x7f4200066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 -- 172.21.15.196:0/287539294 shutdown_connections 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 -- 172.21.15.196:0/287539294 wait complete. 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 10 monclient: monmap: 2024-03-25T19:33:52.068 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 librados: starting msgr at 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.062+0000 7f42067f6640 1 librados: starting objecter 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 Processor -- start 2024-03-25T19:33:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 -- start start 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 librados: setting wanted keys 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 librados: calling monclient init 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient: init 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-c con 0x7f42000a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-a con 0x7f42000b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4200142a20 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200135ec0 con 0x7f42000b4700 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000abf30 con 0x7f4200142a20 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000ade80 con 0x7f42000a96b0 2024-03-25T19:33:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.063+0000 7f42067f6640 10 monclient(hunting): authenticate will time out at 6119.351481s 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41ff7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42000b4700 0x7f42000ae7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.196:41116/0 (socket says 172.21.15.196:41116) 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f4204d93640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4200142a20 0x7f42000acaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51390/0 (socket says 172.21.15.196:51390) 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41ff7fe640 1 -- 172.21.15.196:0/967541452 learned_addr learned my addr 172.21.15.196:0/967541452 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter ms_handle_connect 0x7f4200142a20 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter ms_handle_connect 0x7f42000b4700 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter ms_handle_connect 0x7f42000a96b0 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.064+0000 7f41fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017282693 0 0) 0x7f42000aa1f0 con 0x7f42000b4700 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec004eb0 con 0x7f42000b4700 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411689310 0 0) 0x7f42000abf30 con 0x7f4200142a20 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42000aa1f0 con 0x7f4200142a20 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778013790 0 0) 0x7f42000ade80 con 0x7f42000a96b0 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:33:52.073 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.066+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42000abf30 con 0x7f42000a96b0 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 607721574 0 0) 0x7f42000aa1f0 con 0x7f4200142a20 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42000ade80 con 0x7f4200142a20 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f8001d60 con 0x7f4200142a20 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.075 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898396570 0 0) 0x7f42000abf30 con 0x7f42000a96b0 2024-03-25T19:33:52.076 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42000aa1f0 con 0x7f42000a96b0 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f0001870 con 0x7f42000a96b0 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618965887 0 0) 0x7f41ec004eb0 con 0x7f42000b4700 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42000abf30 con 0x7f42000b4700 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f40017d0 con 0x7f42000b4700 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.071+0000 7f41fcff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1598968964 0 0) 0x7f42000ade80 con 0x7f4200142a20 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 >> v1:172.21.15.29:6790/0 conn(0x7f42000a96b0 legacy=0x7f420013aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 >> v1:172.21.15.29:6789/0 conn(0x7f42000b4700 legacy=0x7f42000ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 1 monclient: found mon.b 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000ae0f0 con 0x7f4200142a20 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f41fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 10 monclient: _renew_subs 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42000ae3d0 con 0x7f4200142a20 2024-03-25T19:33:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 10 monclient: _renew_subs 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42001e34b0 con 0x7f4200142a20 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.074+0000 7f42067f6640 1 librados: init done 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.075+0000 7f42067f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.075+0000 7f42067f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.075+0000 7f42067f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.081 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.075+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42000abd00 con 0x7f4200142a20 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f8004cc0 con 0x7f4200142a20 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41f8004e30 con 0x7f4200142a20 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 10 monclient: dump: 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.194 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41f8005320 con 0x7f4200142a20 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3699149866 0 0) 0x7f41f8006ae0 con 0x7f4200142a20 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fe7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 10 client.4147.objecter ms_dispatch 0x7f4200000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.193+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41f8018de0 con 0x7f4200142a20 2024-03-25T19:33:52.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.194+0000 7f41fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.194+0000 7f41fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:52.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.268+0000 7f42067f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cd5f7e4a-4d35-42f5-96a3-e362c57fd75a", "id": 6}] 2024-03-25T19:33:52.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.268+0000 7f42067f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cd5f7e4a-4d35-42f5-96a3-e362c57fd75a", "id": 6}] 2024-03-25T19:33:52.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.268+0000 7f42067f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.268+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cd5f7e4a-4d35-42f5-96a3-e362c57fd75a", "id": 6} v 0) v1 -- 0x7f42000ab3a0 con 0x7f4200142a20 2024-03-25T19:33:52.310 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:52.311+0000 7f3ab5e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:33:52.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.328+0000 7f41fcff9640 1 -- 172.21.15.196:0/967541452 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd5f7e4a-4d35-42f5-96a3-e362c57fd75a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 501272078 0 1113811849) 0x7f41f8006620 con 0x7f4200142a20 2024-03-25T19:33:52.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.328+0000 7f41fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cd5f7e4a-4d35-42f5-96a3-e362c57fd75a", "id": 6}] 2024-03-25T19:33:52.328 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.328+0000 7f41fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:52.328 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-03-25T19:33:52.329 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 10 librados: watch_flush enter 2024-03-25T19:33:52.329 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 10 librados: watch_flush exit 2024-03-25T19:33:52.329 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 10 client.4147.objecter successfully canceled tick 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 10 monclient: shutdown 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 >> v1:172.21.15.196:6789/0 conn(0x7f4200142a20 legacy=0x7f42000acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 shutdown_connections 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 >> 172.21.15.196:0/967541452 conn(0x7f42000668e0 msgr2=0x7f42000b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 shutdown_connections 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 -- 172.21.15.196:0/967541452 wait complete. 2024-03-25T19:33:52.330 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.329+0000 7f42067f6640 1 librados: shutdown 2024-03-25T19:33:52.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new b025c754-d72e-42ca-951d-695966331614 7 2024-03-25T19:33:52.378 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:52.378+0000 7f9936441040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.437+0000 7ff2a1d44640 10 monclient: get_monmap_and_config 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.437+0000 7ff2a1d44640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.437+0000 7ff2a1d44640 10 monclient: monmap: 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:33:52.438870+0000 2024-03-25T19:33:52.437 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:33:52.438870+0000 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:52.438 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.438+0000 7ff2a1d44640 1 Processor -- start 2024-03-25T19:33:52.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.438+0000 7ff2a1d44640 1 -- start start 2024-03-25T19:33:52.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.438+0000 7ff2a1d44640 10 monclient: init 2024-03-25T19:33:52.439 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c0b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c142a20 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-c con 0x7ff29c0a96b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0a9d60 con 0x7ff29c0b4700 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c13ff10 con 0x7ff29c0a96b0 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0aa1f0 con 0x7ff29c142a20 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff2a1d44640 10 monclient(hunting): authenticate will time out at 6119.726959s 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff29bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff29c0a96b0 0x7ff29c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51402/0 (socket says 172.21.15.196:51402) 2024-03-25T19:33:52.440 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.439+0000 7ff29bfff640 1 -- 172.21.15.196:0/2939175082 learned_addr learned my addr 172.21.15.196:0/2939175082 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610297935 0 0) 0x7ff29c0a9d60 con 0x7ff29c0b4700 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:33:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff288003520 con 0x7ff29c0b4700 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130834393 0 0) 0x7ff29c13ff10 con 0x7ff29c0a96b0 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c0a9d60 con 0x7ff29c0a96b0 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369986639 0 0) 0x7ff29c0aa1f0 con 0x7ff29c142a20 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:33:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.442+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c13ff10 con 0x7ff29c142a20 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492131578 0 0) 0x7ff29c0a9d60 con 0x7ff29c0a96b0 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 >> v1:172.21.15.29:6790/0 conn(0x7ff29c142a20 legacy=0x7ff29c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 >> v1:172.21.15.29:6789/0 conn(0x7ff29c0b4700 legacy=0x7ff29c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 monclient: found mon.noname-c 2024-03-25T19:33:52.445 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c0b4ae0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff2940019f0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff29affd640 10 monclient: dump: 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.446 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff2a1d44640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:33:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.445+0000 7ff2a1d44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff294001b50 con 0x7ff29c0a96b0 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 1 -- 172.21.15.196:0/2939175082 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff294001fa0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 4 set_mon_vals no callback set 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff29affd640 10 monclient: dump: 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 10 monclient: shutdown 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 1 -- 172.21.15.196:0/2939175082 >> v1:172.21.15.196:6789/0 conn(0x7ff29c0a96b0 legacy=0x7ff29c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 1 -- 172.21.15.196:0/2939175082 shutdown_connections 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.447+0000 7ff2a1d44640 1 -- 172.21.15.196:0/2939175082 >> 172.21.15.196:0/2939175082 conn(0x7ff29c0668e0 msgr2=0x7ff29c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 1 -- 172.21.15.196:0/2939175082 shutdown_connections 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 1 -- 172.21.15.196:0/2939175082 wait complete. 2024-03-25T19:33:52.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 10 monclient: monmap: 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 1 librados: starting msgr at 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.448+0000 7ff2a1d44640 1 librados: starting objecter 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 Processor -- start 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 -- start start 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 librados: setting wanted keys 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 librados: calling monclient init 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient: init 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-c con 0x7ff29c0a96b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c0b4700 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c142a20 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0b1850 con 0x7ff29c0b4700 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0addc0 con 0x7ff29c142a20 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0ae0a0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.449+0000 7ff2a1d44640 10 monclient(hunting): authenticate will time out at 6119.737419s 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff29bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff29c142a20 0x7ff29c1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51410/0 (socket says 172.21.15.196:51410) 2024-03-25T19:33:52.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff29bfff640 1 -- 172.21.15.196:0/1717561552 learned_addr learned my addr 172.21.15.196:0/1717561552 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter ms_handle_connect 0x7ff29c142a20 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter ms_handle_connect 0x7ff29c0b4700 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter ms_handle_connect 0x7ff29c0a96b0 2024-03-25T19:33:52.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.450+0000 7ff298ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356261758 0 0) 0x7ff29c0addc0 con 0x7ff29c142a20 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27c004eb0 con 0x7ff29c142a20 2024-03-25T19:33:52.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936514076 0 0) 0x7ff29c0b1850 con 0x7ff29c0b4700 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c0addc0 con 0x7ff29c0b4700 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971082035 0 0) 0x7ff29c0ae0a0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.452+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c0b1850 con 0x7ff29c0a96b0 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.456+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141800246 0 0) 0x7ff27c004eb0 con 0x7ff29c142a20 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.456+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff29c0ae0a0 con 0x7ff29c142a20 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff294001fe0 con 0x7ff29c142a20 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.458 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618345235 0 0) 0x7ff29c0addc0 con 0x7ff29c0b4700 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff27c004eb0 con 0x7ff29c0b4700 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff2840018b0 con 0x7ff29c0b4700 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.459 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448756899 0 0) 0x7ff29c0b1850 con 0x7ff29c0a96b0 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff29c0addc0 con 0x7ff29c0a96b0 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff28c001230 con 0x7ff29c0a96b0 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:52.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.457+0000 7ff298ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.461 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3656102072 0 0) 0x7ff29c0ae0a0 con 0x7ff29c142a20 2024-03-25T19:33:52.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 >> v1:172.21.15.29:6790/0 conn(0x7ff29c0a96b0 legacy=0x7ff29c0ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 >> v1:172.21.15.29:6789/0 conn(0x7ff29c0b4700 legacy=0x7ff29c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 1 monclient: found mon.b 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c0ae380 con 0x7ff29c142a20 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff298ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 10 monclient: _renew_subs 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c0abd20 con 0x7ff29c142a20 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:52.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 10 monclient: _renew_subs 2024-03-25T19:33:52.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c0aca80 con 0x7ff29c142a20 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.460+0000 7ff2a1d44640 1 librados: init done 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.461+0000 7ff2a1d44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.461+0000 7ff2a1d44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.461+0000 7ff2a1d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.461+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c0ad350 con 0x7ff29c142a20 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff294004e10 con 0x7ff29c142a20 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff294005120 con 0x7ff29c142a20 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 10 monclient: dump: 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 3 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff29a7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff294005610 con 0x7ff29c142a20 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 715331310 0 0) 0x7ff294006d20 con 0x7ff29c142a20 2024-03-25T19:33:52.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 10 client.4153.objecter ms_dispatch 0x7ff29c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:33:52.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:33:52.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:33:52.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.464+0000 7ff298ff9640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T19:33:52.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.558+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff294006770 con 0x7ff29c142a20 2024-03-25T19:33:52.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.558+0000 7ff298ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.558+0000 7ff298ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:52.599 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:52.600+0000 7f3ab5e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:33:52.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.645+0000 7ff2a1d44640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b025c754-d72e-42ca-951d-695966331614", "id": 7}] 2024-03-25T19:33:52.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.645+0000 7ff2a1d44640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b025c754-d72e-42ca-951d-695966331614", "id": 7}] 2024-03-25T19:33:52.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.645+0000 7ff2a1d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.645+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b025c754-d72e-42ca-951d-695966331614", "id": 7} v 0) v1 -- 0x7ff29c04f730 con 0x7ff29c142a20 2024-03-25T19:33:52.653 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:52.654+0000 7f3ab5e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:33:52.678 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:52.678+0000 7f9936441040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:33:52.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.707+0000 7ff298ff9640 1 -- 172.21.15.196:0/1717561552 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b025c754-d72e-42ca-951d-695966331614", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 273559701 0 1371665406) 0x7ff294006910 con 0x7ff29c142a20 2024-03-25T19:33:52.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.707+0000 7ff298ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b025c754-d72e-42ca-951d-695966331614", "id": 7}] 2024-03-25T19:33:52.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.707+0000 7ff298ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:52.707 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-03-25T19:33:52.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 10 librados: watch_flush enter 2024-03-25T19:33:52.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 10 librados: watch_flush exit 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 10 client.4153.objecter successfully canceled tick 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 10 monclient: shutdown 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 >> v1:172.21.15.196:6789/0 conn(0x7ff29c142a20 legacy=0x7ff29c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 shutdown_connections 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 >> 172.21.15.196:0/1717561552 conn(0x7ff29c0668e0 msgr2=0x7ff29c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 shutdown_connections 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.708+0000 7ff2a1d44640 1 -- 172.21.15.196:0/1717561552 wait complete. 2024-03-25T19:33:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-25T19:33:52.709+0000 7ff2a1d44640 1 librados: shutdown 2024-03-25T19:33:52.722 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.722 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:33:52.722 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:33:52.725 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:33:52.725 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.725 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:33:52.725 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:33:52.727 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:33:52.727 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.727 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:33:52.727 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:33:52.729 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:33:52.729 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.730 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:33:52.730 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:33:52.732 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:33:52.732 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.732 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:33:52.732 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:33:52.738 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:52.738+0000 7f9936441040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:33:52.740 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:33:52.740 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.740 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:33:52.740 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:33:52.742 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:33:52.742 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.742 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:33:52.742 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:33:52.744 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:33:52.745 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:33:52.745 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:33:52.745 DEBUG:teuthology.orchestra.run.smithi196:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:33:52.747 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:33:52.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:52.789 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:52.790+0000 7f3ab5e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:33:52.871 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-25T19:33:52.872+0000 7f9936441040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:33:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.906+0000 7fdb9250e640 10 monclient: get_monmap_and_config 2024-03-25T19:33:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.906+0000 7fdb9250e640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.906+0000 7fdb9250e640 10 monclient: monmap: 2024-03-25T19:33:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:33:52.907125+0000 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:33:52.907125+0000 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:52.906 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.907+0000 7fdb9250e640 1 Processor -- start 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.907+0000 7fdb9250e640 1 -- start start 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.907+0000 7fdb9250e640 10 monclient: init 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-a con 0x7fdb8c0a9920 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb8c0b45a0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb8c146450 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c06b100 con 0x7fdb8c0a9920 2024-03-25T19:33:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c0b2940 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c13fec0 con 0x7fdb8c146450 2024-03-25T19:33:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.908+0000 7fdb9250e640 10 monclient(hunting): authenticate will time out at 6109.661459s 2024-03-25T19:33:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.909+0000 7fdb8b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdb8c0b45a0 0x7fdb8c145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:59256/0 (socket says 172.21.15.29:59256) 2024-03-25T19:33:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.909+0000 7fdb8b7fe640 1 -- 172.21.15.29:0/3783112496 learned_addr learned my addr 172.21.15.29:0/3783112496 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451147993 0 0) 0x7fdb8c0b2940 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb74003520 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505747888 0 0) 0x7fdb8c13fec0 con 0x7fdb8c146450 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.912+0000 7fdb8affd640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:33:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.913+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb8c0b2940 con 0x7fdb8c146450 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.915+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3551613163 0 0) 0x7fdb74003520 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.915+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 >> v1:172.21.15.29:6790/0 conn(0x7fdb8c146450 legacy=0x7fdb8c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.915+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 >> v1:172.21.15.29:6789/0 conn(0x7fdb8c0a9920 legacy=0x7fdb8c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 1 monclient: found mon.noname-c 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c140050 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb800015c0 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb8affd640 10 monclient: dump: 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb9250e640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.916+0000 7fdb9250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb80001720 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 1 -- 172.21.15.29:0/3783112496 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb80001990 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb8affd640 10 monclient: dump: 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb9250e640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.918+0000 7fdb9250e640 4 set_mon_vals no callback set 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 10 monclient: shutdown 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 1 -- 172.21.15.29:0/3783112496 >> v1:172.21.15.196:6789/0 conn(0x7fdb8c0b45a0 legacy=0x7fdb8c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 1 -- 172.21.15.29:0/3783112496 shutdown_connections 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 1 -- 172.21.15.29:0/3783112496 >> 172.21.15.29:0/3783112496 conn(0x7fdb8c0b7f40 msgr2=0x7fdb8c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 1 -- 172.21.15.29:0/3783112496 shutdown_connections 2024-03-25T19:33:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.919+0000 7fdb9250e640 1 -- 172.21.15.29:0/3783112496 wait complete. 2024-03-25T19:33:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.920+0000 7fdb9250e640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.920+0000 7fdb9250e640 10 monclient: monmap: 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 librados: starting msgr at 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 librados: starting objecter 2024-03-25T19:33:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 Processor -- start 2024-03-25T19:33:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 -- start start 2024-03-25T19:33:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 librados: setting wanted keys 2024-03-25T19:33:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 1 librados: calling monclient init 2024-03-25T19:33:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.921+0000 7fdb9250e640 10 monclient: init 2024-03-25T19:33:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-a con 0x7fdb8c0a9920 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb8c0b45a0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb8c146450 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c140260 con 0x7fdb8c0a9920 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.922+0000 7fdb9250e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c144400 con 0x7fdb8c146450 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb9250e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c144040 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb9250e640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb9250e640 10 monclient(hunting): authenticate will time out at 6109.675945s 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb8b7fe640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fdb8c0b45a0 0x7fdb8c1d7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:49580/0 (socket says 172.21.15.29:49580) 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb8b7fe640 1 -- 172.21.15.29:0/227754652 learned_addr learned my addr 172.21.15.29:0/227754652 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb88ff9640 10 client.?.objecter ms_handle_connect 0x7fdb8c0b45a0 2024-03-25T19:33:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.923+0000 7fdb88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.924+0000 7fdb88ff9640 10 client.?.objecter ms_handle_connect 0x7fdb8c146450 2024-03-25T19:33:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.924+0000 7fdb88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552331628 0 0) 0x7fdb8c144400 con 0x7fdb8c146450 2024-03-25T19:33:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78004eb0 con 0x7fdb8c146450 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204380332 0 0) 0x7fdb8c144040 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:33:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.926+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb8c144400 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851690935 0 0) 0x7fdb78004eb0 con 0x7fdb8c146450 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb8c144040 con 0x7fdb8c146450 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb84001bd0 con 0x7fdb8c146450 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.931+0000 7fdb88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.933+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218405269 0 0) 0x7fdb8c144400 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.933+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb78004eb0 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.934+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb80001a30 con 0x7fdb8c0b45a0 2024-03-25T19:33:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.934+0000 7fdb88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.934+0000 7fdb88ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.934+0000 7fdb88ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.934 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568832336 0 0) 0x7fdb8c144040 con 0x7fdb8c146450 2024-03-25T19:33:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 >> v1:172.21.15.29:6790/0 conn(0x7fdb8c0b45a0 legacy=0x7fdb8c1d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 >> v1:172.21.15.29:6789/0 conn(0x7fdb8c0a9920 legacy=0x7fdb8c143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 1 monclient: found mon.b 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c1441d0 con 0x7fdb8c146450 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.937+0000 7fdb88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 10 monclient: _renew_subs 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c144790 con 0x7fdb8c146450 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 10 monclient: _renew_subs 2024-03-25T19:33:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c1e2bb0 con 0x7fdb8c146450 2024-03-25T19:33:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.938+0000 7fdb9250e640 1 librados: init done 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb84002080 con 0x7fdb8c146450 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb84003b10 con 0x7fdb8c146450 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb88ff9640 10 monclient: dump: 2024-03-25T19:33:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.939+0000 7fdb8a7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.940+0000 7fdb9250e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.940+0000 7fdb9250e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.940+0000 7fdb9250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.940+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c005080 con 0x7fdb8c146450 2024-03-25T19:33:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdb84004000 con 0x7fdb8c146450 2024-03-25T19:33:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fdb84005900 con 0x7fdb8c146450 2024-03-25T19:33:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 10 client.4159.objecter ms_dispatch 0x7fdb8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 3 client.4159.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 3 client.4159.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:52.942+0000 7fdb88ff9640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T19:33:53.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.040+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdb84005b10 con 0x7fdb8c146450 2024-03-25T19:33:53.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.040+0000 7fdb88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.040+0000 7fdb88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.166+0000 7fdb9250e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.166+0000 7fdb9250e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.166+0000 7fdb9250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:33:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.166+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb6c005330 con 0x7fdb8c146450 2024-03-25T19:33:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.168+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1821465853 0 0) 0x7fdb840052e0 con 0x7fdb8c146450 2024-03-25T19:33:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.432+0000 7fdb88ff9640 1 -- 172.21.15.29:0/227754652 <== mon.1 v1:172.21.15.196:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fdb84014bc0 con 0x7fdb8c146450 2024-03-25T19:33:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.432+0000 7fdb88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.432+0000 7fdb88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:53.432 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-25T19:33:53.432 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.433+0000 7fdb9250e640 10 librados: watch_flush enter 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.433+0000 7fdb9250e640 10 librados: watch_flush exit 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.433+0000 7fdb9250e640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.433+0000 7fdb9250e640 10 client.4159.objecter successfully canceled tick 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 10 monclient: shutdown 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 >> v1:172.21.15.196:6789/0 conn(0x7fdb8c146450 legacy=0x7fdb8c1d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 shutdown_connections 2024-03-25T19:33:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 >> 172.21.15.29:0/227754652 conn(0x7fdb8c0b7f40 msgr2=0x7fdb8c0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 shutdown_connections 2024-03-25T19:33:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 -- 172.21.15.29:0/227754652 wait complete. 2024-03-25T19:33:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-25T19:33:53.434+0000 7fdb9250e640 1 librados: shutdown 2024-03-25T19:33:53.448 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:33:53.448 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:33:54.252 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.408+0000 7f0b726c1640 10 monclient: get_monmap_and_config 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.408+0000 7f0b726c1640 10 monclient: build_initial_monmap 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.408+0000 7f0b726c1640 10 monclient: monmap: 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:33:54.409448+0000 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:33:54.409448+0000 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:54.408 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:54.409 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:54.409 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:33:54.409 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:33:54.409 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.409+0000 7f0b726c1640 1 Processor -- start 2024-03-25T19:33:54.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.409+0000 7f0b726c1640 1 -- start start 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.409+0000 7f0b726c1640 10 monclient: init 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.410+0000 7f0b726c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.410+0000 7f0b726c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.410+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c142520 con 0x7f0b6c1430c0 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c0b28a0 con 0x7f0b6c13f710 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c0b2a30 con 0x7f0b6c0a98f0 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:54.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.411+0000 7f0b726c1640 10 monclient(hunting): authenticate will time out at 6111.164370s 2024-03-25T19:33:54.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.488+0000 7f0b6bfff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f0b6c0a98f0 0x7f0b6c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:49678/0 (socket says 172.21.15.29:49678) 2024-03-25T19:33:54.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.488+0000 7f0b6bfff640 1 -- 172.21.15.29:0/1100612184 learned_addr learned my addr 172.21.15.29:0/1100612184 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573369623 0 0) 0x7f0b6c142520 con 0x7f0b6c1430c0 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:33:54.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.613+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b54003520 con 0x7f0b6c1430c0 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786320291 0 0) 0x7f0b6c0b2a30 con 0x7f0b6c0a98f0 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 10 monclient(hunting): my global_id is 4172 2024-03-25T19:33:54.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.628+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b6c142520 con 0x7f0b6c0a98f0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899192150 0 0) 0x7f0b54003520 con 0x7f0b6c1430c0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 >> v1:172.21.15.29:6790/0 conn(0x7f0b6c0a98f0 legacy=0x7f0b6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 >> v1:172.21.15.196:6789/0 conn(0x7f0b6c13f710 legacy=0x7f0b6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 monclient: found mon.noname-a 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0b44c0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: _finish_auth 0 2024-03-25T19:33:54.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b64001a10 con 0x7f0b6c1430c0 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.636+0000 7f0b6affd640 10 monclient: dump: 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:54.637 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:54.638 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.637+0000 7f0b726c1640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:33:54.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.637+0000 7f0b726c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:54.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b64001b70 con 0x7f0b6c1430c0 2024-03-25T19:33:54.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:54.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 1 -- 172.21.15.29:0/1100612184 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b64001f80 con 0x7f0b6c1430c0 2024-03-25T19:33:54.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b726c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:54.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b726c1640 4 set_mon_vals no callback set 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.641+0000 7f0b6affd640 10 monclient: dump: 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:54.641 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 10 monclient: shutdown 2024-03-25T19:33:54.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:54.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 1 -- 172.21.15.29:0/1100612184 >> v1:172.21.15.29:6789/0 conn(0x7f0b6c1430c0 legacy=0x7f0b6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 1 -- 172.21.15.29:0/1100612184 shutdown_connections 2024-03-25T19:33:54.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 1 -- 172.21.15.29:0/1100612184 >> 172.21.15.29:0/1100612184 conn(0x7f0b6c0b7ea0 msgr2=0x7f0b6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:54.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 1 -- 172.21.15.29:0/1100612184 shutdown_connections 2024-03-25T19:33:54.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.642+0000 7f0b726c1640 1 -- 172.21.15.29:0/1100612184 wait complete. 2024-03-25T19:33:54.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.643+0000 7f0b726c1640 10 monclient: build_initial_monmap 2024-03-25T19:33:54.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.643+0000 7f0b726c1640 10 monclient: monmap: 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.644+0000 7f0b726c1640 1 librados: starting msgr at 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.644+0000 7f0b726c1640 1 librados: starting objecter 2024-03-25T19:33:54.644 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.644+0000 7f0b726c1640 1 Processor -- start 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.645+0000 7f0b726c1640 1 -- start start 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.645+0000 7f0b726c1640 1 librados: setting wanted keys 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.645+0000 7f0b726c1640 1 librados: calling monclient init 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.645+0000 7f0b726c1640 10 monclient: init 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:33:54.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c0b7c30 con 0x7f0b6c1430c0 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.646+0000 7f0b726c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c137520 con 0x7f0b6c0a98f0 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.647+0000 7f0b726c1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c1d7ba0 con 0x7f0b6c13f710 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.647+0000 7f0b726c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:54.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.647+0000 7f0b726c1640 10 monclient(hunting): authenticate will time out at 6111.400062s 2024-03-25T19:33:54.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.648+0000 7f0b70c5e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b6c1430c0 0x7f0b6c1e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47326/0 (socket says 172.21.15.29:47326) 2024-03-25T19:33:54.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.648+0000 7f0b70c5e640 1 -- 172.21.15.29:0/701843243 learned_addr learned my addr 172.21.15.29:0/701843243 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:33:54.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.648+0000 7f0b68ff9640 10 client.?.objecter ms_handle_connect 0x7f0b6c1430c0 2024-03-25T19:33:54.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.648+0000 7f0b68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062014451 0 0) 0x7f0b6c0b7c30 con 0x7f0b6c1430c0 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:33:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.651+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58004eb0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3769525797 0 0) 0x7f0b58004eb0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b6c0b7c30 con 0x7f0b6c1430c0 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b64001f80 con 0x7f0b6c1430c0 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.659 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.659+0000 7f0b68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.670+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1799149290 0 0) 0x7f0b6c0b7c30 con 0x7f0b6c1430c0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.670+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 >> v1:172.21.15.29:6790/0 conn(0x7f0b6c13f710 legacy=0x7f0b6c133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.670+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 >> v1:172.21.15.196:6789/0 conn(0x7f0b6c0a98f0 legacy=0x7f0b6c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 1 monclient: found mon.a 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c136950 con 0x7f0b6c1430c0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:54.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 10 monclient: _renew_subs 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c136ba0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 10 monclient: _renew_subs 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6c137010 con 0x7f0b6c1430c0 2024-03-25T19:33:54.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.671+0000 7f0b726c1640 1 librados: init done 2024-03-25T19:33:54.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.672+0000 7f0b726c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.672+0000 7f0b726c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.672+0000 7f0b726c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:54.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.672+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6c0627b0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:54.677+0000 7f3a5627d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:33:54.688 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:33:54.689+0000 7f3a5627d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:33:54.700 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b64003c20 con 0x7f0b6c1430c0 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b640020e0 con 0x7f0b6c1430c0 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b68ff9640 10 monclient: dump: 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:33:54.701 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:33:54.702 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:33:54.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.701+0000 7f0b6a7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:54.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.702+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1875230740 0 0) 0x7f0b64004130 con 0x7f0b6c1430c0 2024-03-25T19:33:54.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.703+0000 7f0b6bfff640 10 monclient: get_auth_request con 0x7f0b5806da90 auth_method 0 2024-03-25T19:33:54.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.703+0000 7f0b68ff9640 10 client.4170.objecter ms_handle_connect 0x7f0b5806da90 2024-03-25T19:33:54.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.706+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f0b64054c40 con 0x7f0b6c1430c0 2024-03-25T19:33:54.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.706+0000 7f0b68ff9640 10 client.4170.objecter ms_dispatch 0x7f0b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:54.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.706+0000 7f0b68ff9640 3 client.4170.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:54.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.706+0000 7f0b68ff9640 3 client.4170.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:54.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:54.706+0000 7f0b68ff9640 20 client.4170.objecter dump_active .. 0 homeless 2024-03-25T19:33:55.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.151+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b64050750 con 0x7f0b6c1430c0 2024-03-25T19:33:55.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.151+0000 7f0b68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:55.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.151+0000 7f0b68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:55.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.521+0000 7f0b726c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:55.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.521+0000 7f0b726c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:55.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.521+0000 7f0b726c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:33:55.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.521+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b6c1376b0 con 0x7f0b6c1430c0 2024-03-25T19:33:55.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.652+0000 7f0b68ff9640 1 -- 172.21.15.29:0/701843243 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f0b64052680 con 0x7f0b6c1430c0 2024-03-25T19:33:55.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.652+0000 7f0b68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:55.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.652+0000 7f0b68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:55.652 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:33:55.652 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:55.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 10 librados: watch_flush enter 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 10 librados: watch_flush exit 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 20 client.4170.objecter shutdown clearing up homeless session... 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 10 client.4170.objecter successfully canceled tick 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 >> v1:172.21.15.29:6800/2314944989 conn(0x7f0b5806da90 legacy=0x7f0b5806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 10 monclient: shutdown 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 >> v1:172.21.15.29:6789/0 conn(0x7f0b6c1430c0 legacy=0x7f0b6c1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 shutdown_connections 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.655+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 >> 172.21.15.29:0/701843243 conn(0x7f0b6c0b7ea0 msgr2=0x7f0b6c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.656+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 shutdown_connections 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.656+0000 7f0b726c1640 1 -- 172.21.15.29:0/701843243 wait complete. 2024-03-25T19:33:55.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:33:55.656+0000 7f0b726c1640 1 librados: shutdown 2024-03-25T19:33:55.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:59.551 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:06.746 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:33:59.602 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:06.774 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:33:59.625 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:06.821 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:33:59.673 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:06.845 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:33:59.778 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:06.968 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:33:59.840 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:07.029 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.039 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:07.210 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.073 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:07.259 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.098 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:07.268 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.134 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:07.321 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.166 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-25T19:34:00.160+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.228 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-25T19:34:00.222+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.232 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:07.428 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.266 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:07.473 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.293 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:07.488 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.327 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:07.534 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.386 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-25T19:34:00.380+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.433 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:07.617 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.493 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:07.677 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.626 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:07.822 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.645 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-25T19:34:00.639+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.682 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:07.854 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.713 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-03-25T19:34:00.708+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.766 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:07.961 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.804 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:07.976 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.843 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:08.033 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:00.843 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-03-25T19:34:00.838+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:00.885 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-03-25T19:34:00.880+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:01.068 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-03-25T19:34:01.060+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:01.087 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:08.283 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.101 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:08.271 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.131 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:08.303 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.189 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:08.375 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.213 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:08.403 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.341 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:08.536 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.357 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:08.552 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.357 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:08.564 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.401 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:08.573 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.464 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:08.635 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.537 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:08.720 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.561 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:08.747 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.566 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:08.756 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.621 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:08.817 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.666 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:08.838 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.709 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:08.904 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.733 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:08.940 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.818 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:08.989 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.856 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:09.046 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.893 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:09.089 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.908 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:09.092 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.918 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:09.104 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:01.930 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:09.102 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.023 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:09.218 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.095 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:09.302 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.104 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:09.275 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.149 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:09.339 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.162 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:09.357 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.197 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:09.369 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.206 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:09.392 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.272 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:09.455 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.299 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:09.495 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.383 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:09.590 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.396 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:09.567 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.434 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:09.630 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.439 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:09.629 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.463 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:09.635 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.477 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:02.511 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:09.697 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.568+0000 7f7909074640 10 monclient: get_monmap_and_config 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.568+0000 7f7909074640 10 monclient: build_initial_monmap 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.568+0000 7f7909074640 10 monclient: monmap: 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:02.569461+0000 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:02.569461+0000 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:02.568 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:02.569 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 1 Processor -- start 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 1 -- start start 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 10 monclient: init 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.569+0000 7f7909074640 10 monclient(hunting): picked mon.noname-c con 0x7f79040a96e0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 10 monclient(hunting): picked mon.noname-b con 0x7f79040b1b10 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 10 monclient(hunting): picked mon.noname-a con 0x7f7904146060 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7904140040 con 0x7f7904146060 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040b1fa0 con 0x7f79040a96e0 2024-03-25T19:34:02.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040b24c0 con 0x7f79040b1b10 2024-03-25T19:34:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f7909074640 10 monclient(hunting): authenticate will time out at 6119.322914s 2024-03-25T19:34:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f790359e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7904146060 0x7f790414dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33732/0 (socket says 172.21.15.29:33732) 2024-03-25T19:34:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f78fbfff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f79040b1b10 0x7f7904145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:55580/0 (socket says 172.21.15.29:55580) 2024-03-25T19:34:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.570+0000 7f790359e640 1 -- 172.21.15.29:0/2413563293 learned_addr learned my addr 172.21.15.29:0/2413563293 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:02.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145344865 0 0) 0x7f79040b1fa0 con 0x7f79040a96e0 2024-03-25T19:34:02.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f0003520 con 0x7f79040a96e0 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757500524 0 0) 0x7f79040b24c0 con 0x7f79040b1b10 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79040b1fa0 con 0x7f79040b1b10 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091676284 0 0) 0x7f7904140040 con 0x7f7904146060 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 10 monclient(hunting): my global_id is 4197 2024-03-25T19:34:02.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.573+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79040b24c0 con 0x7f7904146060 2024-03-25T19:34:02.574 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:09.756 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465467341 0 0) 0x7f78f0003520 con 0x7f79040a96e0 2024-03-25T19:34:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 >> v1:172.21.15.29:6790/0 conn(0x7f79040b1b10 legacy=0x7f7904145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 >> v1:172.21.15.29:6789/0 conn(0x7f7904146060 legacy=0x7f790414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: _finish_hunting 0 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 monclient: found mon.noname-c 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040b15b0 con 0x7f79040a96e0 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: _finish_auth 0 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: _check_auth_tickets 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78f40015c0 con 0x7f79040a96e0 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f790259c640 10 monclient: dump: 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.576 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.577 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.577 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.577 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.577 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.577 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.578 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.578 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.578 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f7909074640 5 monclient: authenticate success, global_id 4198 2024-03-25T19:34:02.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.576+0000 7f7909074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:02.578 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:09.771 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78f4001720 con 0x7f79040a96e0 2024-03-25T19:34:02.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 1 -- 172.21.15.29:0/2413563293 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78f4001b30 con 0x7f79040a96e0 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f790259c640 10 monclient: dump: 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.582 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f7909074640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.577+0000 7f7909074640 4 set_mon_vals no callback set 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 10 monclient: shutdown 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 1 -- 172.21.15.29:0/2413563293 >> v1:172.21.15.196:6789/0 conn(0x7f79040a96e0 legacy=0x7f79040b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 1 -- 172.21.15.29:0/2413563293 shutdown_connections 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 1 -- 172.21.15.29:0/2413563293 >> 172.21.15.29:0/2413563293 conn(0x7f79040b7ea0 msgr2=0x7f79040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 1 -- 172.21.15.29:0/2413563293 shutdown_connections 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.578+0000 7f7909074640 1 -- 172.21.15.29:0/2413563293 wait complete. 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 10 monclient: build_initial_monmap 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 10 monclient: monmap: 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:02.583 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 librados: starting msgr at 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 librados: starting objecter 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 Processor -- start 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 -- start start 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 librados: setting wanted keys 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 1 librados: calling monclient init 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.579+0000 7f7909074640 10 monclient: init 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:02.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient(hunting): picked mon.noname-a con 0x7f79040a96e0 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient(hunting): picked mon.noname-b con 0x7f7904146060 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient(hunting): picked mon.noname-c con 0x7f790413aa40 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790413aeb0 con 0x7f79040a96e0 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040ae7e0 con 0x7f7904146060 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040aeac0 con 0x7f790413aa40 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.580+0000 7f7909074640 10 monclient(hunting): authenticate will time out at 6119.333390s 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f7902d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f79040a96e0 0x7f79040ad4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33740/0 (socket says 172.21.15.29:33740) 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f7902d9d640 1 -- 172.21.15.29:0/143657796 learned_addr learned my addr 172.21.15.29:0/143657796 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter ms_handle_connect 0x7f79040a96e0 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter ms_handle_connect 0x7f790413aa40 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter ms_handle_connect 0x7f7904146060 2024-03-25T19:34:02.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.581+0000 7f78fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221901943 0 0) 0x7f79040ae7e0 con 0x7f7904146060 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e4004eb0 con 0x7f7904146060 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141578743 0 0) 0x7f790413aeb0 con 0x7f79040a96e0 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.583+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79040ae7e0 con 0x7f79040a96e0 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372474359 0 0) 0x7f79040aeac0 con 0x7f790413aa40 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.584+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790413aeb0 con 0x7f790413aa40 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261422909 0 0) 0x7f78e4004eb0 con 0x7f7904146060 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79040aeac0 con 0x7f7904146060 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78ec001a30 con 0x7f7904146060 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.587+0000 7f78fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:02.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210364231 0 0) 0x7f79040ae7e0 con 0x7f79040a96e0 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78e4004eb0 con 0x7f79040a96e0 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78f4001a30 con 0x7f79040a96e0 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.588+0000 7f78fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.592 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2799315497 0 0) 0x7f790413aeb0 con 0x7f790413aa40 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79040ae7e0 con 0x7f790413aa40 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78fc001bd0 con 0x7f790413aa40 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.589+0000 7f78fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.593 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1968281175 0 0) 0x7f79040aeac0 con 0x7f7904146060 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 >> v1:172.21.15.29:6790/0 conn(0x7f790413aa40 legacy=0x7f79040af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 >> v1:172.21.15.29:6789/0 conn(0x7f79040a96e0 legacy=0x7f79040ad4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 1 monclient: found mon.b 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040af360 con 0x7f7904146060 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f78fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:02.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f7909074640 5 monclient: authenticate success, global_id 4201 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f7909074640 10 monclient: _renew_subs 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f7909074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.590+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790413b060 con 0x7f7904146060 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.591+0000 7f7909074640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.591+0000 7f7909074640 10 monclient: _renew_subs 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.591+0000 7f7909074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.591+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79040aff90 con 0x7f7904146060 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.591+0000 7f7909074640 1 librados: init done 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.592+0000 7f7909074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.592+0000 7f7909074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.592+0000 7f7909074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:02.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.592+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79040a9ac0 con 0x7f7904146060 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78ec001c50 con 0x7f7904146060 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f78ec003a60 con 0x7f7904146060 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f78fb7fe640 10 monclient: dump: 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:02.605 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:02.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.605+0000 7f7901d9b640 4 set_mon_vals no callback set 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.606+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f78ec003c80 con 0x7f7904146060 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f7902d9d640 10 monclient: get_auth_request con 0x7f78e406e0b0 auth_method 0 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 10 client.4201.objecter ms_handle_connect 0x7f78e406e0b0 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f78ec054790 con 0x7f7904146060 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 10 client.4201.objecter ms_dispatch 0x7f7904000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:02.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.607+0000 7f78fb7fe640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T19:34:02.648 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:09.855 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.692 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:09.863 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.703 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:09.898 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.716 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:09.905 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.733 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:09.905 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.812 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:09.999 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.845+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78ec0502a0 con 0x7f7904146060 2024-03-25T19:34:02.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.845+0000 7f78fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:02.845+0000 7f78fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:02.847 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.043 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.878 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.061 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.939 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.146 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.988 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:10.178 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:02.992 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:10.163 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.047+0000 7f7909074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:03.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.047+0000 7f7909074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:03.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.047+0000 7f7909074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:03.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.047+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79040af4f0 con 0x7f7904146060 2024-03-25T19:34:03.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.072+0000 7f78fb7fe640 1 -- 172.21.15.29:0/143657796 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f78ec0521d0 con 0x7f7904146060 2024-03-25T19:34:03.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.072+0000 7f78fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:03.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.072+0000 7f78fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:03.071 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:03.071 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 10 librados: watch_flush enter 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 10 librados: watch_flush exit 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 10 client.4201.objecter successfully canceled tick 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 >> v1:172.21.15.29:6800/2314944989 conn(0x7f78e406e0b0 legacy=0x7f78e4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 10 monclient: shutdown 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 >> v1:172.21.15.196:6789/0 conn(0x7f7904146060 legacy=0x7f790413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 shutdown_connections 2024-03-25T19:34:03.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 >> 172.21.15.29:0/143657796 conn(0x7f79040b7ea0 msgr2=0x7f790413de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:03.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 shutdown_connections 2024-03-25T19:34:03.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 -- 172.21.15.29:0/143657796 wait complete. 2024-03-25T19:34:03.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:03.074+0000 7f7909074640 1 librados: shutdown 2024-03-25T19:34:03.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:03.108 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.294 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.121 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.317 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.192 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.376 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.202 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:10.398 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.214 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:10.386 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.239 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.446 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.256 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:10.451 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.267 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:10.439 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.290 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:10.480 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.294 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:10.465 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.386 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.581 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.403 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.589 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.484 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.668 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.526 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.733 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.589 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:10.759 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.650 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.845 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.692 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.878 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.773 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.957 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.788 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:10.978 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.816 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.023 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:03.843 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:11.032 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.077 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.260 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.130 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:11.326 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.175 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:11.345 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.186 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:11.382 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.243 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:11.413 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.255 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.441 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.311 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.497 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.367 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.574 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.423 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.630 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.732 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.916 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:04.788 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.972 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:08.398 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:15.593 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:08.629 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:15.801 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.189 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:16.379 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.664 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:16.860 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.688 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:16.859 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.763 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:16.949 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.871 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:17.078 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:09.889 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:09.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.994+0000 7f3f8ee2e640 10 monclient: get_monmap_and_config 2024-03-25T19:34:09.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.994+0000 7f3f8ee2e640 10 monclient: build_initial_monmap 2024-03-25T19:34:09.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.994+0000 7f3f8ee2e640 10 monclient: monmap: 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:09.995703+0000 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:09.995703+0000 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:09.995 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.995+0000 7f3f8ee2e640 1 Processor -- start 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.995+0000 7f3f8ee2e640 1 -- start start 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.995+0000 7f3f8ee2e640 10 monclient: init 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f880b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f880b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f8814be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880af940 con 0x7f3f8814be90 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880b0a70 con 0x7f3f880b1730 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880b0d50 con 0x7f3f880b4d60 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:09.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.996+0000 7f3f8ee2e640 10 monclient(hunting): authenticate will time out at 6126.749397s 2024-03-25T19:34:09.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.997+0000 7f3f87fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3f880b1730 0x7f3f880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:45678/0 (socket says 172.21.15.29:45678) 2024-03-25T19:34:09.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:09.997+0000 7f3f87fff640 1 -- 172.21.15.29:0/1760390214 learned_addr learned my addr 172.21.15.29:0/1760390214 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804319252 0 0) 0x7f3f880b0a70 con 0x7f3f880b1730 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f78003520 con 0x7f3f880b1730 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968149242 0 0) 0x7f3f880b0d50 con 0x7f3f880b4d60 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.002+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f880b0a70 con 0x7f3f880b4d60 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509840268 0 0) 0x7f3f880af940 con 0x7f3f8814be90 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:34:10.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.003+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f880b0d50 con 0x7f3f8814be90 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3883662942 0 0) 0x7f3f880b0a70 con 0x7f3f880b4d60 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 >> v1:172.21.15.196:6789/0 conn(0x7f3f880b1730 legacy=0x7f3f880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 >> v1:172.21.15.29:6789/0 conn(0x7f3f8814be90 legacy=0x7f3f8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 monclient: found mon.noname-b 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:34:10.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880b1030 con 0x7f3f880b4d60 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f7c0015f0 con 0x7f3f880b4d60 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f877fe640 10 monclient: dump: 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.007 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f8ee2e640 5 monclient: authenticate success, global_id 4205 2024-03-25T19:34:10.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.006+0000 7f3f8ee2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:10.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f7c001750 con 0x7f3f880b4d60 2024-03-25T19:34:10.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:10.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f8ee2e640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:10.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f8ee2e640 4 set_mon_vals no callback set 2024-03-25T19:34:10.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 1 -- 172.21.15.29:0/1760390214 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f7c001b60 con 0x7f3f880b4d60 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f877fe640 10 monclient: dump: 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f8ee2e640 10 monclient: shutdown 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f8ee2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.009+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1760390214 >> v1:172.21.15.29:6790/0 conn(0x7f3f880b4d60 legacy=0x7f3f8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1760390214 shutdown_connections 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1760390214 >> 172.21.15.29:0/1760390214 conn(0x7f3f880617a0 msgr2=0x7f3f88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1760390214 shutdown_connections 2024-03-25T19:34:10.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1760390214 wait complete. 2024-03-25T19:34:10.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 10 monclient: build_initial_monmap 2024-03-25T19:34:10.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.010+0000 7f3f8ee2e640 10 monclient: monmap: 2024-03-25T19:34:10.013 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:10.013 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:10.013 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 librados: starting msgr at 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 librados: starting objecter 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 Processor -- start 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 -- start start 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 librados: setting wanted keys 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 1 librados: calling monclient init 2024-03-25T19:34:10.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.011+0000 7f3f8ee2e640 10 monclient: init 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f880b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f880b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8814be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f881d7620 con 0x7f3f880b1730 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f881d84a0 con 0x7f3f8814be90 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.012+0000 7f3f8ee2e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f88141e30 con 0x7f3f880b4d60 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f8ee2e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f8ee2e640 10 monclient(hunting): authenticate will time out at 6126.765948s 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f87fff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f3f880b4d60 0x7f3f88142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:55646/0 (socket says 172.21.15.29:55646) 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f87fff640 1 -- 172.21.15.29:0/1131283539 learned_addr learned my addr 172.21.15.29:0/1131283539 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f857fa640 10 client.?.objecter ms_handle_connect 0x7f3f880b4d60 2024-03-25T19:34:10.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:10.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f857fa640 10 client.?.objecter ms_handle_connect 0x7f3f8814be90 2024-03-25T19:34:10.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.013+0000 7f3f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:10.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.014+0000 7f3f857fa640 10 client.?.objecter ms_handle_connect 0x7f3f880b1730 2024-03-25T19:34:10.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.014+0000 7f3f857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596473393 0 0) 0x7f3f881d84a0 con 0x7f3f8814be90 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c004eb0 con 0x7f3f8814be90 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47084091 0 0) 0x7f3f88141e30 con 0x7f3f880b4d60 2024-03-25T19:34:10.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f881d84a0 con 0x7f3f880b4d60 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225682503 0 0) 0x7f3f881d7620 con 0x7f3f880b1730 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:34:10.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.016+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f88141e30 con 0x7f3f880b1730 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424804437 0 0) 0x7f3f88141e30 con 0x7f3f880b1730 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c005620 con 0x7f3f880b1730 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f7c001a60 con 0x7f3f880b1730 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 10 monclient(hunting): dump: 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:10.023 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.024 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.021+0000 7f3f857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487935678 0 0) 0x7f3f6c004eb0 con 0x7f3f8814be90 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c005cd0 con 0x7f3f8814be90 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f80001b00 con 0x7f3f8814be90 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.022+0000 7f3f857fa640 10 monclient(hunting): dump: 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:10.025 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:10.026 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:10.026 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:10.026 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.024+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 484601738 0 0) 0x7f3f881d84a0 con 0x7f3f880b4d60 2024-03-25T19:34:10.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.024+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c004eb0 con 0x7f3f880b4d60 2024-03-25T19:34:10.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4178570899 0 0) 0x7f3f6c005620 con 0x7f3f880b1730 2024-03-25T19:34:10.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 >> v1:172.21.15.29:6790/0 conn(0x7f3f880b4d60 legacy=0x7f3f88142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 >> v1:172.21.15.196:6789/0 conn(0x7f3f8814be90 legacy=0x7f3f88142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 1 monclient: found mon.a 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f881d7870 con 0x7f3f880b1730 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 5 monclient: authenticate success, global_id 4206 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 10 monclient: _renew_subs 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f88140e60 con 0x7f3f880b1730 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 10 monclient: _renew_subs 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:34:10.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f88143530 con 0x7f3f880b1730 2024-03-25T19:34:10.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.025+0000 7f3f8ee2e640 1 librados: init done 2024-03-25T19:34:10.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.026+0000 7f3f8ee2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:10.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.026+0000 7f3f8ee2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:10.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.026+0000 7f3f8ee2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:34:10.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.026+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f881d7ab0 con 0x7f3f880b1730 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f7c001050 con 0x7f3f880b1730 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f7c001a60 con 0x7f3f880b1730 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f857fa640 10 monclient: dump: 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.076 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.076+0000 7f3f86ffd640 4 set_mon_vals no callback set 2024-03-25T19:34:10.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.077+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f3f7c019710 con 0x7f3f880b1730 2024-03-25T19:34:10.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.078+0000 7f3f87fff640 10 monclient: get_auth_request con 0x7f3f6c06e7d0 auth_method 0 2024-03-25T19:34:10.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.079+0000 7f3f857fa640 10 client.4206.objecter ms_handle_connect 0x7f3f6c06e7d0 2024-03-25T19:34:10.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.188+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f3f7c0530f0 con 0x7f3f880b1730 2024-03-25T19:34:10.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.188+0000 7f3f857fa640 10 client.4206.objecter ms_dispatch 0x7f3f88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:10.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.188+0000 7f3f857fa640 3 client.4206.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:10.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.188+0000 7f3f857fa640 3 client.4206.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:10.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.188+0000 7f3f857fa640 20 client.4206.objecter dump_active .. 0 homeless 2024-03-25T19:34:10.302 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:17.486 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:10.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.349+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f7c04ea30 con 0x7f3f880b1730 2024-03-25T19:34:10.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.349+0000 7f3f857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:10.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.349+0000 7f3f857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:10.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.630+0000 7f3f8ee2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.630+0000 7f3f8ee2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.630+0000 7f3f8ee2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:34:10.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.630+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f880627b0 con 0x7f3f880b1730 2024-03-25T19:34:10.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.647+0000 7f3f857fa640 1 -- 172.21.15.29:0/1131283539 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f3f7c0018c0 con 0x7f3f880b1730 2024-03-25T19:34:10.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.647+0000 7f3f857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.647+0000 7f3f857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:10.647 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:10.647 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 10 librados: watch_flush enter 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 10 librados: watch_flush exit 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 20 client.4206.objecter shutdown clearing up homeless session... 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 10 client.4206.objecter successfully canceled tick 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 >> v1:172.21.15.29:6800/2314944989 conn(0x7f3f6c06e7d0 legacy=0x7f3f6c070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 10 monclient: shutdown 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 >> v1:172.21.15.29:6789/0 conn(0x7f3f880b1730 legacy=0x7f3f881d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 shutdown_connections 2024-03-25T19:34:10.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 >> 172.21.15.29:0/1131283539 conn(0x7f3f880617a0 msgr2=0x7f3f8813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:10.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 shutdown_connections 2024-03-25T19:34:10.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 -- 172.21.15.29:0/1131283539 wait complete. 2024-03-25T19:34:10.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:10.650+0000 7f3f8ee2e640 1 librados: shutdown 2024-03-25T19:34:10.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:14.221 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:21.416 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:14.224 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:21.420 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:14.622 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:21.794 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:14.626 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:21.798 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:14.890 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:22.079 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:14.894 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:22.083 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.494 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:22.665 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.500 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:22.671 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.637 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:22.824 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.641 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:22.828 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.737 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:22.944 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:15.741 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:22.949 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:16.082 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:23.277 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:16.086 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:23.281 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:16.210 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:23.393 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:16.213 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:23.397 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:17.470 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.560+0000 7f1d40cd9640 10 monclient: get_monmap_and_config 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.560+0000 7f1d40cd9640 10 monclient: build_initial_monmap 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.560+0000 7f1d40cd9640 10 monclient: monmap: 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:17.561581+0000 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:17.561581+0000 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:17.560 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:17.561 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:17.561 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.561+0000 7f1d40cd9640 1 Processor -- start 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.561+0000 7f1d40cd9640 1 -- start start 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.561+0000 7f1d40cd9640 10 monclient: init 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0af940 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0b0a70 con 0x7f1d3c14be90 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0b0d50 con 0x7f1d3c0b1730 2024-03-25T19:34:17.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:17.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d40cd9640 10 monclient(hunting): authenticate will time out at 6134.315279s 2024-03-25T19:34:17.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d3a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d3c0b4d60 0x7f1d3c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37480/0 (socket says 172.21.15.29:37480) 2024-03-25T19:34:17.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d39d9b640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f1d3c0b1730 0x7f1d3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:46596/0 (socket says 172.21.15.29:46596) 2024-03-25T19:34:17.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.562+0000 7f1d39d9b640 1 -- 172.21.15.29:0/4106255390 learned_addr learned my addr 172.21.15.29:0/4106255390 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.565+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355281097 0 0) 0x7f1d3c0b0a70 con 0x7f1d3c14be90 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.565+0000 7f1d3959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.565+0000 7f1d3959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.565+0000 7f1d3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.565+0000 7f1d3959a640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d24003520 con 0x7f1d3c14be90 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728471458 0 0) 0x7f1d3c0af940 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:34:17.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.566+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c0b0a70 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534153033 0 0) 0x7f1d3c0b0d50 con 0x7f1d3c0b1730 2024-03-25T19:34:17.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d24004a30 con 0x7f1d3c0b1730 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327027928 0 0) 0x7f1d3c0b0a70 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 >> v1:172.21.15.29:6790/0 conn(0x7f1d3c0b1730 legacy=0x7f1d3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.568+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 >> v1:172.21.15.196:6789/0 conn(0x7f1d3c14be90 legacy=0x7f1d3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: _finish_hunting 0 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 1 monclient: found mon.noname-a 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0b1030 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: _finish_auth 0 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: _check_auth_tickets 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d2c0015f0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d3959a640 10 monclient: dump: 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d40cd9640 5 monclient: authenticate success, global_id 4209 2024-03-25T19:34:17.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.569+0000 7f1d40cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:17.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d2c001750 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:17.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 1 -- 172.21.15.29:0/4106255390 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d2c001ba0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d3959a640 10 monclient: dump: 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d40cd9640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.571+0000 7f1d40cd9640 4 set_mon_vals no callback set 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 10 monclient: shutdown 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 1 -- 172.21.15.29:0/4106255390 >> v1:172.21.15.29:6789/0 conn(0x7f1d3c0b4d60 legacy=0x7f1d3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 1 -- 172.21.15.29:0/4106255390 shutdown_connections 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 1 -- 172.21.15.29:0/4106255390 >> 172.21.15.29:0/4106255390 conn(0x7f1d3c0617a0 msgr2=0x7f1d3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 1 -- 172.21.15.29:0/4106255390 shutdown_connections 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.572+0000 7f1d40cd9640 1 -- 172.21.15.29:0/4106255390 wait complete. 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.573+0000 7f1d40cd9640 10 monclient: build_initial_monmap 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.573+0000 7f1d40cd9640 10 monclient: monmap: 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:17.574 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:17.575 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.573+0000 7f1d40cd9640 1 librados: starting msgr at 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.573+0000 7f1d40cd9640 1 librados: starting objecter 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.574+0000 7f1d40cd9640 1 Processor -- start 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.574+0000 7f1d40cd9640 1 -- start start 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.574+0000 7f1d40cd9640 1 librados: setting wanted keys 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.574+0000 7f1d40cd9640 1 librados: calling monclient init 2024-03-25T19:34:17.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.574+0000 7f1d40cd9640 10 monclient: init 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c1352e0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c1418a0 con 0x7f1d3c14be90 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c141b80 con 0x7f1d3c0b1730 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d40cd9640 10 monclient(hunting): authenticate will time out at 6134.328173s 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d39d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d3c0b4d60 0x7f1d3c1344c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37486/0 (socket says 172.21.15.29:37486) 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d39d9b640 1 -- 172.21.15.29:0/3574235871 learned_addr learned my addr 172.21.15.29:0/3574235871 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d2affd640 10 client.?.objecter ms_handle_connect 0x7f1d3c0b1730 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.575+0000 7f1d2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.576+0000 7f1d2affd640 10 client.?.objecter ms_handle_connect 0x7f1d3c0b4d60 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.576+0000 7f1d2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.576+0000 7f1d2affd640 10 client.?.objecter ms_handle_connect 0x7f1d3c14be90 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.576+0000 7f1d2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508430505 0 0) 0x7f1d3c1352e0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d1c004eb0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931607072 0 0) 0x7f1d3c1418a0 con 0x7f1d3c14be90 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:34:17.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.578+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c1352e0 con 0x7f1d3c14be90 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091124429 0 0) 0x7f1d3c141b80 con 0x7f1d3c0b1730 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:34:17.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.579+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c1418a0 con 0x7f1d3c0b1730 2024-03-25T19:34:17.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465608345 0 0) 0x7f1d3c1352e0 con 0x7f1d3c14be90 2024-03-25T19:34:17.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d3c141b80 con 0x7f1d3c14be90 2024-03-25T19:34:17.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d34001a70 con 0x7f1d3c14be90 2024-03-25T19:34:17.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): dump: 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873942460 0 0) 0x7f1d1c004eb0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d3c1352e0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d300019f0 con 0x7f1d3c0b4d60 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.583+0000 7f1d2affd640 10 monclient(hunting): dump: 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.585 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2154782869 0 0) 0x7f1d3c1418a0 con 0x7f1d3c0b1730 2024-03-25T19:34:17.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d1c004eb0 con 0x7f1d3c0b1730 2024-03-25T19:34:17.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d2c001a60 con 0x7f1d3c0b1730 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.585+0000 7f1d2affd640 10 monclient(hunting): dump: 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1389977673 0 0) 0x7f1d3c141b80 con 0x7f1d3c14be90 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 >> v1:172.21.15.29:6790/0 conn(0x7f1d3c0b1730 legacy=0x7f1d3c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 >> v1:172.21.15.29:6789/0 conn(0x7f1d3c0b4d60 legacy=0x7f1d3c1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 1 monclient: found mon.b 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c141e60 con 0x7f1d3c14be90 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 10 monclient: _finish_auth 0 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 10 monclient: _renew_subs 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c142140 con 0x7f1d3c14be90 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:17.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 10 monclient: _renew_subs 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c142560 con 0x7f1d3c14be90 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.586+0000 7f1d40cd9640 1 librados: init done 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.588+0000 7f1d40cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.588+0000 7f1d40cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.588+0000 7f1d40cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.588+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c0627b0 con 0x7f1d3c14be90 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d34001f60 con 0x7f1d3c14be90 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1d34003b30 con 0x7f1d3c14be90 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d2affd640 10 monclient: dump: 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:17.609 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.609+0000 7f1d38d99640 4 set_mon_vals no callback set 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.610+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f1d34019b00 con 0x7f1d3c14be90 2024-03-25T19:34:17.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.610+0000 7f1d3a59c640 10 monclient: get_auth_request con 0x7f1d1c06e0b0 auth_method 0 2024-03-25T19:34:17.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.611+0000 7f1d2affd640 10 client.4213.objecter ms_handle_connect 0x7f1d1c06e0b0 2024-03-25T19:34:17.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.613+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f1d34053750 con 0x7f1d3c14be90 2024-03-25T19:34:17.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.613+0000 7f1d2affd640 10 client.4213.objecter ms_dispatch 0x7f1d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:17.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.613+0000 7f1d2affd640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:17.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.613+0000 7f1d2affd640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:17.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.613+0000 7f1d2affd640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:34:17.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.849+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d3404f090 con 0x7f1d3c14be90 2024-03-25T19:34:17.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.849+0000 7f1d2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:17.849+0000 7f1d2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:18.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.157+0000 7f1d40cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:18.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.157+0000 7f1d40cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:18.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.157+0000 7f1d40cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:18.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.157+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d3c061b80 con 0x7f1d3c14be90 2024-03-25T19:34:18.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.182+0000 7f1d2affd640 1 -- 172.21.15.29:0/3574235871 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f1d34051190 con 0x7f1d3c14be90 2024-03-25T19:34:18.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.182+0000 7f1d2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:18.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.182+0000 7f1d2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:18.182 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:18.182 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:18.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.184+0000 7f1d40cd9640 10 librados: watch_flush enter 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 10 librados: watch_flush exit 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 >> v1:172.21.15.29:6800/2314944989 conn(0x7f1d1c06e0b0 legacy=0x7f1d1c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 10 monclient: shutdown 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 >> v1:172.21.15.196:6789/0 conn(0x7f1d3c14be90 legacy=0x7f1d3c134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 shutdown_connections 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 >> 172.21.15.29:0/3574235871 conn(0x7f1d3c0617a0 msgr2=0x7f1d3c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 shutdown_connections 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 -- 172.21.15.29:0/3574235871 wait complete. 2024-03-25T19:34:18.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:18.185+0000 7f1d40cd9640 1 librados: shutdown 2024-03-25T19:34:18.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:18.767 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:00:25.962 35696== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:19.121 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:00:26.293 35698== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:19.565 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:00:26.755 35697== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:20.082 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:00:27.252 35699== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:20.189 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:27.376 35386== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:20.471 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:27.678 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:20.572 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:27.756 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:20.814 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:28.010 35385== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:25.004 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.143+0000 7f3411f15640 10 monclient: get_monmap_and_config 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.143+0000 7f3411f15640 10 monclient: build_initial_monmap 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.143+0000 7f3411f15640 10 monclient: monmap: 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:25.145633+0000 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:25.145633+0000 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:25.144 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:25.145 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:25.145 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.144+0000 7f3411f15640 1 Processor -- start 2024-03-25T19:34:25.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.144+0000 7f3411f15640 1 -- start start 2024-03-25T19:34:25.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.144+0000 7f3411f15640 10 monclient: init 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.145+0000 7f3411f15640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.145+0000 7f3411f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.145+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-c con 0x7f340c0b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.145+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-a con 0x7f340c0b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-b con 0x7f340c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0af940 con 0x7f340c0b1730 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0b0a70 con 0x7f340c0b4d60 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0b0d50 con 0x7f340c14be90 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f3411f15640 10 monclient(hunting): authenticate will time out at 6141.900162s 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f340bfff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f340c14be90 0x7f340c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:46608/0 (socket says 172.21.15.29:46608) 2024-03-25T19:34:25.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.146+0000 7f340bfff640 1 -- 172.21.15.29:0/3740833840 learned_addr learned my addr 172.21.15.29:0/3740833840 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667154380 0 0) 0x7f340c0af940 con 0x7f340c0b1730 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:34:25.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f4003520 con 0x7f340c0b1730 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136826899 0 0) 0x7f340c0b0a70 con 0x7f340c0b4d60 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.150+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340c0af940 con 0x7f340c0b4d60 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486092000 0 0) 0x7f340c0b0d50 con 0x7f340c14be90 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:34:25.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.151+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340c0b0a70 con 0x7f340c14be90 2024-03-25T19:34:25.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139075812 0 0) 0x7f340c0af940 con 0x7f340c0b4d60 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 >> v1:172.21.15.29:6790/0 conn(0x7f340c14be90 legacy=0x7f340c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 >> v1:172.21.15.29:6789/0 conn(0x7f340c0b1730 legacy=0x7f340c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.155+0000 7f340a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0b1030 con 0x7f340c0b4d60 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f33fc0015f0 con 0x7f340c0b4d60 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f3411f15640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:34:25.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f3411f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f340a7fc640 10 monclient: dump: 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.156+0000 7f3411f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33fc001810 con 0x7f340c0b4d60 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 1 -- 172.21.15.29:0/3740833840 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f33fc001ba0 con 0x7f340c0b4d60 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:25.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f340a7fc640 10 monclient: dump: 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f3411f15640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.158+0000 7f3411f15640 4 set_mon_vals no callback set 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 10 monclient: shutdown 2024-03-25T19:34:25.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 1 -- 172.21.15.29:0/3740833840 >> v1:172.21.15.196:6789/0 conn(0x7f340c0b4d60 legacy=0x7f340c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 1 -- 172.21.15.29:0/3740833840 shutdown_connections 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 1 -- 172.21.15.29:0/3740833840 >> 172.21.15.29:0/3740833840 conn(0x7f340c0617a0 msgr2=0x7f340c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 1 -- 172.21.15.29:0/3740833840 shutdown_connections 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.159+0000 7f3411f15640 1 -- 172.21.15.29:0/3740833840 wait complete. 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.160+0000 7f3411f15640 10 monclient: build_initial_monmap 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.160+0000 7f3411f15640 10 monclient: monmap: 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:25.164 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.160+0000 7f3411f15640 1 librados: starting msgr at 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.160+0000 7f3411f15640 1 librados: starting objecter 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.161+0000 7f3411f15640 1 Processor -- start 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.161+0000 7f3411f15640 1 -- start start 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.161+0000 7f3411f15640 1 librados: setting wanted keys 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.161+0000 7f3411f15640 1 librados: calling monclient init 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.161+0000 7f3411f15640 10 monclient: init 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:25.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-b con 0x7f340c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:25.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-c con 0x7f340c0b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:25.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient(hunting): picked mon.noname-a con 0x7f340c14be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c137fc0 con 0x7f340c14be90 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c13c340 con 0x7f340c0b1730 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c06ebe0 con 0x7f340c0b4d60 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f3411f15640 10 monclient(hunting): authenticate will time out at 6141.916269s 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f340b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f340c0b1730 0x7f340c139180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:43288/0 (socket says 172.21.15.29:43288) 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.162+0000 7f340b7fe640 1 -- 172.21.15.29:0/65457127 learned_addr learned my addr 172.21.15.29:0/65457127 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter ms_handle_connect 0x7f340c0b1730 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter ms_handle_connect 0x7f340c0b4d60 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter ms_handle_connect 0x7f340c14be90 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.163+0000 7f33f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841531789 0 0) 0x7f340c13c340 con 0x7f340c0b1730 2024-03-25T19:34:25.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f8004eb0 con 0x7f340c0b1730 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111555173 0 0) 0x7f340c137fc0 con 0x7f340c14be90 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340c13c340 con 0x7f340c14be90 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300025917 0 0) 0x7f340c06ebe0 con 0x7f340c0b4d60 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.166+0000 7f33f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.167+0000 7f33f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.167+0000 7f33f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.171 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.167+0000 7f33f3fff640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:34:25.171 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.167+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340c137fc0 con 0x7f340c0b4d60 2024-03-25T19:34:25.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.172+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615131287 0 0) 0x7f33f8004eb0 con 0x7f340c0b1730 2024-03-25T19:34:25.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f340c06ebe0 con 0x7f340c0b1730 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f33fc0015f0 con 0x7f340c0b1730 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 10 monclient(hunting): dump: 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:25.174 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:25.175 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.173+0000 7f33f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.174+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270517813 0 0) 0x7f340c137fc0 con 0x7f340c0b4d60 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.174+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f8004eb0 con 0x7f340c0b4d60 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.174+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24007707 0 0) 0x7f340c13c340 con 0x7f340c14be90 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.175+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f340c137fc0 con 0x7f340c14be90 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.175+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3404001ac0 con 0x7f340c14be90 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.175+0000 7f33f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.175+0000 7f33f3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.175+0000 7f33f3fff640 10 monclient(hunting): dump: 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.176 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.177 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.177 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:25.177 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:25.177 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:25.177 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.178+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270530631 0 0) 0x7f340c06ebe0 con 0x7f340c0b1730 2024-03-25T19:34:25.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.178+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 >> v1:172.21.15.29:6790/0 conn(0x7f340c0b4d60 legacy=0x7f340c1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.178+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 >> v1:172.21.15.29:6789/0 conn(0x7f340c14be90 legacy=0x7f340c1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 1 monclient: found mon.b 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c06eec0 con 0x7f340c0b1730 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f33f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 10 monclient: _renew_subs 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c13be20 con 0x7f340c0b1730 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:25.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 10 monclient: _renew_subs 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c06dea0 con 0x7f340c0b1730 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.179+0000 7f3411f15640 1 librados: init done 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f3411f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f3411f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f3411f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c13bc40 con 0x7f340c0b1730 2024-03-25T19:34:25.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33fc0038a0 con 0x7f340c0b1730 2024-03-25T19:34:25.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f33f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:25.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.181+0000 7f3409ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:25.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.200+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f33fc003c70 con 0x7f340c0b1730 2024-03-25T19:34:25.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.200+0000 7f33f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.200+0000 7f33f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:25.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.200+0000 7f33f3fff640 10 monclient: dump: 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:25.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.201+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f33fc003e50 con 0x7f340c0b1730 2024-03-25T19:34:25.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.202+0000 7f340affd640 10 monclient: get_auth_request con 0x7f33f806e2f0 auth_method 0 2024-03-25T19:34:25.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.202+0000 7f33f3fff640 10 client.4219.objecter ms_handle_connect 0x7f33f806e2f0 2024-03-25T19:34:25.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.203+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f33fc054960 con 0x7f340c0b1730 2024-03-25T19:34:25.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.203+0000 7f33f3fff640 10 client.4219.objecter ms_dispatch 0x7f340c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:25.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.203+0000 7f33f3fff640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:25.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.203+0000 7f33f3fff640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:25.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.203+0000 7f33f3fff640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:34:25.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.503+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33fc0502a0 con 0x7f340c0b1730 2024-03-25T19:34:25.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.503+0000 7f33f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.503+0000 7f33f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:25.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.822+0000 7f3411f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.822+0000 7f3411f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.822+0000 7f3411f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:25.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.822+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f340c13bfb0 con 0x7f340c0b1730 2024-03-25T19:34:25.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.845+0000 7f33f3fff640 1 -- 172.21.15.29:0/65457127 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f33fc0523a0 con 0x7f340c0b1730 2024-03-25T19:34:25.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.845+0000 7f33f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.845+0000 7f33f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:25.846 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:25.846 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:25.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 10 librados: watch_flush enter 2024-03-25T19:34:25.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 10 librados: watch_flush exit 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 >> v1:172.21.15.29:6800/2314944989 conn(0x7f33f806e2f0 legacy=0x7f33f80707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 10 monclient: shutdown 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.847+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 >> v1:172.21.15.196:6789/0 conn(0x7f340c0b1730 legacy=0x7f340c139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.848+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 shutdown_connections 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.848+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 >> 172.21.15.29:0/65457127 conn(0x7f340c0617a0 msgr2=0x7f340c143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.848+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 shutdown_connections 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.848+0000 7f3411f15640 1 -- 172.21.15.29:0/65457127 wait complete. 2024-03-25T19:34:25.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:25.848+0000 7f3411f15640 1 librados: shutdown 2024-03-25T19:34:25.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:27.272 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-25T19:34:27.269+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:34:27.476 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-25T19:34:27.419+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:34:28.364 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-25T19:34:28.362+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:34:28.592 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-25T19:34:28.582+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:34:29.077 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-03-25T19:34:29.075+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:34:29.589 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-03-25T19:34:29.585+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:34:30.184 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-03-25T19:34:30.180+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:34:30.376 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-03-25T19:34:30.372+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:34:32.664 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.763+0000 7f14dea63640 10 monclient: get_monmap_and_config 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.763+0000 7f14dea63640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.763+0000 7f14dea63640 10 monclient: monmap: 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:32.765743+0000 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:32.765743+0000 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.764+0000 7f14dea63640 1 Processor -- start 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.764+0000 7f14dea63640 1 -- start start 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.764+0000 7f14dea63640 10 monclient: init 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-a con 0x7f14d8061890 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-c con 0x7f14d813f520 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-b con 0x7f14d813f9a0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d806fe30 con 0x7f14d8061890 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d80ae5b0 con 0x7f14d813f520 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d813fdd0 con 0x7f14d813f9a0 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.765+0000 7f14dea63640 10 monclient(hunting): authenticate will time out at 6149.519316s 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.766+0000 7f14dd000640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f14d813f9a0 0x7f14d8146120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:50396/0 (socket says 172.21.15.29:50396) 2024-03-25T19:34:32.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.766+0000 7f14dd000640 1 -- 172.21.15.29:0/3284430340 learned_addr learned my addr 172.21.15.29:0/3284430340 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:32.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921729227 0 0) 0x7f14d80ae5b0 con 0x7f14d813f520 2024-03-25T19:34:32.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:34:32.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.784+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0003520 con 0x7f14d813f520 2024-03-25T19:34:32.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016833013 0 0) 0x7f14d806fe30 con 0x7f14d8061890 2024-03-25T19:34:32.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 10 monclient(hunting): my global_id is 4245 2024-03-25T19:34:32.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.792+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d80ae5b0 con 0x7f14d8061890 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 759981767 0 0) 0x7f14d80ae5b0 con 0x7f14d8061890 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 >> v1:172.21.15.29:6790/0 conn(0x7f14d813f9a0 legacy=0x7f14d8146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 >> v1:172.21.15.196:6789/0 conn(0x7f14d813f520 legacy=0x7f14d8145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8151bd0 con 0x7f14d8061890 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14dea63640 5 monclient: authenticate success, global_id 4245 2024-03-25T19:34:32.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f14cc001670 con 0x7f14d8061890 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14dea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14d6ffd640 10 monclient: dump: 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:32.802 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.800+0000 7f14dea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:32.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14cc0017d0 con 0x7f14d8061890 2024-03-25T19:34:32.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:32.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 1 -- 172.21.15.29:0/3284430340 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f14cc001ba0 con 0x7f14d8061890 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14d6ffd640 10 monclient: dump: 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14dea63640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:32.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.806+0000 7f14dea63640 4 set_mon_vals no callback set 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 10 monclient: shutdown 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 1 -- 172.21.15.29:0/3284430340 >> v1:172.21.15.29:6789/0 conn(0x7f14d8061890 legacy=0x7f14d80adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 1 -- 172.21.15.29:0/3284430340 shutdown_connections 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 1 -- 172.21.15.29:0/3284430340 >> 172.21.15.29:0/3284430340 conn(0x7f14d80b0cc0 msgr2=0x7f14d80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 1 -- 172.21.15.29:0/3284430340 shutdown_connections 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.807+0000 7f14dea63640 1 -- 172.21.15.29:0/3284430340 wait complete. 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 10 monclient: monmap: 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:32.809 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 librados: starting msgr at 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 librados: starting objecter 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 Processor -- start 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 -- start start 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 librados: setting wanted keys 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 1 librados: calling monclient init 2024-03-25T19:34:32.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.808+0000 7f14dea63640 10 monclient: init 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-a con 0x7f14d8061890 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-c con 0x7f14d813f520 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient(hunting): picked mon.noname-b con 0x7f14d813f9a0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d8132220 con 0x7f14d8061890 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d8131fa0 con 0x7f14d813f9a0 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d8144bf0 con 0x7f14d813f520 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.809+0000 7f14dea63640 10 monclient(hunting): authenticate will time out at 6149.563830s 2024-03-25T19:34:32.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.820+0000 7f14d7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f14d8061890 0x7f14d8140070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45368/0 (socket says 172.21.15.29:45368) 2024-03-25T19:34:32.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.820+0000 7f14d7fff640 1 -- 172.21.15.29:0/2115574215 learned_addr learned my addr 172.21.15.29:0/2115574215 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:32.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.820+0000 7f14d4ff9640 10 client.?.objecter ms_handle_connect 0x7f14d8061890 2024-03-25T19:34:32.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.820+0000 7f14d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.829+0000 7f14d4ff9640 10 client.?.objecter ms_handle_connect 0x7f14d813f9a0 2024-03-25T19:34:32.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.829+0000 7f14d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588320706 0 0) 0x7f14d8131fa0 con 0x7f14d813f9a0 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.841+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c4004eb0 con 0x7f14d813f9a0 2024-03-25T19:34:32.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248160963 0 0) 0x7f14d8151bd0 con 0x7f14d8061890 2024-03-25T19:34:32.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 10 monclient(hunting): my global_id is 4248 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.845+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8131fa0 con 0x7f14d8061890 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.847+0000 7f14d4ff9640 10 client.?.objecter ms_handle_connect 0x7f14d813f520 2024-03-25T19:34:32.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.847+0000 7f14d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1607556181 0 0) 0x7f14c4004eb0 con 0x7f14d813f9a0 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d8151bd0 con 0x7f14d813f9a0 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f14d0001b00 con 0x7f14d813f9a0 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:32.858 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:32.859 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.858+0000 7f14d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775653117 0 0) 0x7f14d8144bf0 con 0x7f14d813f520 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:34:32.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.866+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c4004eb0 con 0x7f14d813f520 2024-03-25T19:34:32.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3595016071 0 0) 0x7f14d8131fa0 con 0x7f14d8061890 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d8144bf0 con 0x7f14d8061890 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f14cc001a00 con 0x7f14d8061890 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.869+0000 7f14d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:32.870 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:32.871 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:32.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.870+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419506984 0 0) 0x7f14d8151bd0 con 0x7f14d813f9a0 2024-03-25T19:34:32.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.870+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 >> v1:172.21.15.29:6790/0 conn(0x7f14d813f520 legacy=0x7f14d8131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 >> v1:172.21.15.29:6789/0 conn(0x7f14d8061890 legacy=0x7f14d8140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 1 monclient: found mon.b 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8144ed0 con 0x7f14d813f9a0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 5 monclient: authenticate success, global_id 4249 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 10 monclient: _renew_subs 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d8143ca0 con 0x7f14d813f9a0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 10 monclient: _renew_subs 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d8140780 con 0x7f14d813f9a0 2024-03-25T19:34:32.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.871+0000 7f14dea63640 1 librados: init done 2024-03-25T19:34:32.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.872+0000 7f14dea63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.872+0000 7f14dea63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.872+0000 7f14dea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:32.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:32.872+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d80627b0 con 0x7f14d813f9a0 2024-03-25T19:34:33.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14d0001f70 con 0x7f14d813f9a0 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f14d0004d00 con 0x7f14d813f9a0 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d4ff9640 10 monclient: dump: 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:33.085 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:33.086 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:33.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.084+0000 7f14d67fc640 4 set_mon_vals no callback set 2024-03-25T19:34:33.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.150+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f14d001acd0 con 0x7f14d813f9a0 2024-03-25T19:34:33.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.151+0000 7f14d7fff640 10 monclient: get_auth_request con 0x7f14c406df10 auth_method 0 2024-03-25T19:34:33.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.151+0000 7f14d4ff9640 10 client.4249.objecter ms_handle_connect 0x7f14c406df10 2024-03-25T19:34:33.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.156+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f14d0054990 con 0x7f14d813f9a0 2024-03-25T19:34:33.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.156+0000 7f14d4ff9640 10 client.4249.objecter ms_dispatch 0x7f14d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:33.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.156+0000 7f14d4ff9640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:33.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.156+0000 7f14d4ff9640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:33.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.156+0000 7f14d4ff9640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-25T19:34:33.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.158+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14d00502d0 con 0x7f14d813f9a0 2024-03-25T19:34:33.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.158+0000 7f14d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:33.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.158+0000 7f14d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:33.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.413+0000 7f14dea63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.413+0000 7f14dea63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.413+0000 7f14dea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:33.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.413+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14d8069f40 con 0x7f14d813f9a0 2024-03-25T19:34:33.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.434+0000 7f14d4ff9640 1 -- 172.21.15.29:0/2115574215 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f14d00523d0 con 0x7f14d813f9a0 2024-03-25T19:34:33.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.434+0000 7f14d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.434+0000 7f14d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:33.435 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:33.435 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 10 librados: watch_flush enter 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 10 librados: watch_flush exit 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 10 client.4249.objecter successfully canceled tick 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 >> v1:172.21.15.29:6800/2314944989 conn(0x7f14c406df10 legacy=0x7f14c40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:33.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 10 monclient: shutdown 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 >> v1:172.21.15.196:6789/0 conn(0x7f14d813f9a0 legacy=0x7f14d8131890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 shutdown_connections 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.436+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 >> 172.21.15.29:0/2115574215 conn(0x7f14d80b0cc0 msgr2=0x7f14d80ada80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.437+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 shutdown_connections 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.437+0000 7f14dea63640 1 -- 172.21.15.29:0/2115574215 wait complete. 2024-03-25T19:34:33.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:33.437+0000 7f14dea63640 1 librados: shutdown 2024-03-25T19:34:33.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:40.257 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.376+0000 7f5db6c8d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.376+0000 7f5db6c8d640 10 monclient: build_initial_monmap 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.376+0000 7f5db6c8d640 10 monclient: monmap: 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:40.377610+0000 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:40.377610+0000 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:40.376 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:40.377 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:40.377 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.377 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.377+0000 7f5db6c8d640 1 Processor -- start 2024-03-25T19:34:40.377 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.377+0000 7f5db6c8d640 1 -- start start 2024-03-25T19:34:40.377 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.377+0000 7f5db6c8d640 10 monclient: init 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f5db00a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f5db013f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f5db01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:40.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.378+0000 7f5db6c8d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db0142520 con 0x7f5db013f710 2024-03-25T19:34:40.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.379+0000 7f5db6c8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00b28a0 con 0x7f5db00a98f0 2024-03-25T19:34:40.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.379+0000 7f5db6c8d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00b2a30 con 0x7f5db01430c0 2024-03-25T19:34:40.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.379+0000 7f5db6c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:40.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.379+0000 7f5db6c8d640 10 monclient(hunting): authenticate will time out at 6157.131920s 2024-03-25T19:34:40.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.384+0000 7f5daffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5db013f710 0x7f5db01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36340/0 (socket says 172.21.15.29:36340) 2024-03-25T19:34:40.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.384+0000 7f5daffff640 1 -- 172.21.15.29:0/2800092164 learned_addr learned my addr 172.21.15.29:0/2800092164 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:40.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419753027 0 0) 0x7f5db0142520 con 0x7f5db013f710 2024-03-25T19:34:40.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:34:40.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.406+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d98003520 con 0x7f5db013f710 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813560824 0 0) 0x7f5d98003520 con 0x7f5db013f710 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 >> v1:172.21.15.29:6790/0 conn(0x7f5db01430c0 legacy=0x7f5db014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 >> v1:172.21.15.196:6789/0 conn(0x7f5db00a98f0 legacy=0x7f5db00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 monclient: found mon.noname-a 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00b44c0 con 0x7f5db013f710 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da4001690 con 0x7f5db013f710 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5daf7fe640 10 monclient: dump: 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5db6c8d640 5 monclient: authenticate success, global_id 4251 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.415+0000 7f5db6c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5da40017f0 con 0x7f5db013f710 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 1 -- 172.21.15.29:0/2800092164 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da4001c00 con 0x7f5db013f710 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5daf7fe640 10 monclient: dump: 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.463 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5db6c8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:40.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5db6c8d640 4 set_mon_vals no callback set 2024-03-25T19:34:40.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5db6c8d640 10 monclient: shutdown 2024-03-25T19:34:40.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5db6c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.464+0000 7f5db6c8d640 1 -- 172.21.15.29:0/2800092164 >> v1:172.21.15.29:6789/0 conn(0x7f5db013f710 legacy=0x7f5db01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.465+0000 7f5db6c8d640 1 -- 172.21.15.29:0/2800092164 shutdown_connections 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.465+0000 7f5db6c8d640 1 -- 172.21.15.29:0/2800092164 >> 172.21.15.29:0/2800092164 conn(0x7f5db00b7ea0 msgr2=0x7f5db00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.465+0000 7f5db6c8d640 1 -- 172.21.15.29:0/2800092164 shutdown_connections 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.465+0000 7f5db6c8d640 1 -- 172.21.15.29:0/2800092164 wait complete. 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 10 monclient: build_initial_monmap 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 10 monclient: monmap: 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:40.466 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 librados: starting msgr at 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 librados: starting objecter 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 Processor -- start 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 -- start start 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 librados: setting wanted keys 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 1 librados: calling monclient init 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.466+0000 7f5db6c8d640 10 monclient: init 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f5db00a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f5db013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f5db01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00abd70 con 0x7f5db00a98f0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db01d7710 con 0x7f5db013f710 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db01d8340 con 0x7f5db01430c0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.467+0000 7f5db6c8d640 10 monclient(hunting): authenticate will time out at 6157.220689s 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.468+0000 7f5db522a640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f5db01430c0 0x7f5db01d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:60848/0 (socket says 172.21.15.29:60848) 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.468+0000 7f5db522a640 1 -- 172.21.15.29:0/247907589 learned_addr learned my addr 172.21.15.29:0/247907589 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.469+0000 7f5dad7fa640 10 client.?.objecter ms_handle_connect 0x7f5db01430c0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.469+0000 7f5dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.502+0000 7f5dad7fa640 10 client.?.objecter ms_handle_connect 0x7f5db013f710 2024-03-25T19:34:40.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.502+0000 7f5dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.522+0000 7f5dad7fa640 10 client.?.objecter ms_handle_connect 0x7f5db00a98f0 2024-03-25T19:34:40.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.522+0000 7f5dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367173055 0 0) 0x7f5db00abd70 con 0x7f5db00a98f0 2024-03-25T19:34:40.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:34:40.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.526+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c004eb0 con 0x7f5db00a98f0 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897523568 0 0) 0x7f5db01d8340 con 0x7f5db01430c0 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:34:40.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.532+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db00abd70 con 0x7f5db01430c0 2024-03-25T19:34:40.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624856877 0 0) 0x7f5db01d7710 con 0x7f5db013f710 2024-03-25T19:34:40.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:34:40.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.534+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db01d8340 con 0x7f5db013f710 2024-03-25T19:34:40.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 88316129 0 0) 0x7f5db01d8340 con 0x7f5db013f710 2024-03-25T19:34:40.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db01d7710 con 0x7f5db013f710 2024-03-25T19:34:40.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da4001b00 con 0x7f5db013f710 2024-03-25T19:34:40.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.554+0000 7f5dad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3384737665 0 0) 0x7f5d9c004eb0 con 0x7f5db00a98f0 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db01d8340 con 0x7f5db00a98f0 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da0001930 con 0x7f5db00a98f0 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.555+0000 7f5dad7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.555 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.556 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.556 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267137211 0 0) 0x7f5db00abd70 con 0x7f5db01430c0 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d9c004eb0 con 0x7f5db01430c0 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da8001bd0 con 0x7f5db01430c0 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.560+0000 7f5dad7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.560 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.561 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2493502587 0 0) 0x7f5db01d7710 con 0x7f5db013f710 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 >> v1:172.21.15.29:6790/0 conn(0x7f5db01430c0 legacy=0x7f5db01d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 >> v1:172.21.15.29:6789/0 conn(0x7f5db00a98f0 legacy=0x7f5db0137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 1 monclient: found mon.b 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db01d84d0 con 0x7f5db013f710 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5dad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 10 monclient: _renew_subs 2024-03-25T19:34:40.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db01d41e0 con 0x7f5db013f710 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 10 monclient: _renew_subs 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db01d5150 con 0x7f5db013f710 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.562+0000 7f5db6c8d640 1 librados: init done 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.563+0000 7f5db6c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.563+0000 7f5db6c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.563+0000 7f5db6c8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:40.564 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.563+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db0074cc0 con 0x7f5db013f710 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5da4004a90 con 0x7f5db013f710 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5da4004da0 con 0x7f5db013f710 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dad7fa640 10 monclient: dump: 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:40.836 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.836+0000 7f5dae7fc640 4 set_mon_vals no callback set 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.837+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f5da401ad70 con 0x7f5db013f710 2024-03-25T19:34:40.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.838+0000 7f5db4a29640 10 monclient: get_auth_request con 0x7f5d9c06e0f0 auth_method 0 2024-03-25T19:34:40.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.838+0000 7f5dad7fa640 10 client.4252.objecter ms_handle_connect 0x7f5d9c06e0f0 2024-03-25T19:34:40.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f5da4054860 con 0x7f5db013f710 2024-03-25T19:34:40.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 10 client.4252.objecter ms_dispatch 0x7f5db0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:40.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:40.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:40.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T19:34:40.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5da4050370 con 0x7f5db013f710 2024-03-25T19:34:40.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:40.843+0000 7f5dad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:41.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.108+0000 7f5db6c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.108+0000 7f5db6c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.108+0000 7f5db6c8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:34:41.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.108+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5db01d78a0 con 0x7f5db013f710 2024-03-25T19:34:41.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.139+0000 7f5dad7fa640 1 -- 172.21.15.29:0/247907589 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f5da40522a0 con 0x7f5db013f710 2024-03-25T19:34:41.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.139+0000 7f5dad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.139+0000 7f5dad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:41.139 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:41.139 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:41.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 10 librados: watch_flush enter 2024-03-25T19:34:41.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 10 librados: watch_flush exit 2024-03-25T19:34:41.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T19:34:41.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 10 client.4252.objecter successfully canceled tick 2024-03-25T19:34:41.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 >> v1:172.21.15.29:6800/2314944989 conn(0x7f5d9c06e0f0 legacy=0x7f5d9c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 10 monclient: shutdown 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.141+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 >> v1:172.21.15.196:6789/0 conn(0x7f5db013f710 legacy=0x7f5db01d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.142+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 shutdown_connections 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.142+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 >> 172.21.15.29:0/247907589 conn(0x7f5db00b7ea0 msgr2=0x7f5db0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.142+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 shutdown_connections 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.142+0000 7f5db6c8d640 1 -- 172.21.15.29:0/247907589 wait complete. 2024-03-25T19:34:41.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:41.142+0000 7f5db6c8d640 1 librados: shutdown 2024-03-25T19:34:41.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:44.345 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:14.509 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.413 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:14.576 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.438 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:14.600 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.476 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:14.639 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.496 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:14.658 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.541 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:14.705 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.641 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:14.804 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.723 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:14.885 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:44.781 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:14.945 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:47.959 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.078+0000 7f2676197640 10 monclient: get_monmap_and_config 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.078+0000 7f2676197640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.078+0000 7f2676197640 10 monclient: monmap: 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:48.079485+0000 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:48.079485+0000 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:48.078 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:48.079 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.079+0000 7f2676197640 1 Processor -- start 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.079+0000 7f2676197640 1 -- start start 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.079+0000 7f2676197640 10 monclient: init 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient(hunting): picked mon.noname-c con 0x7f26700a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient(hunting): picked mon.noname-a con 0x7f267013f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient(hunting): picked mon.noname-b con 0x7f26701430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2670142520 con 0x7f267013f710 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26700b28a0 con 0x7f26700a98f0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26700b2a30 con 0x7f26701430c0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.080+0000 7f2676197640 10 monclient(hunting): authenticate will time out at 6164.833502s 2024-03-25T19:34:48.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.092+0000 7f266f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f26700a98f0 0x7f26700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:53748/0 (socket says 172.21.15.29:53748) 2024-03-25T19:34:48.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.092+0000 7f266f7fe640 1 -- 172.21.15.29:0/4091435448 learned_addr learned my addr 172.21.15.29:0/4091435448 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:48.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634706411 0 0) 0x7f26700b28a0 con 0x7f26700a98f0 2024-03-25T19:34:48.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:34:48.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.104+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2658003520 con 0x7f26700a98f0 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813474711 0 0) 0x7f26700b2a30 con 0x7f26701430c0 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:34:48.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.105+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26700b28a0 con 0x7f26701430c0 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034326397 0 0) 0x7f26700b28a0 con 0x7f26701430c0 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 >> v1:172.21.15.196:6789/0 conn(0x7f26700a98f0 legacy=0x7f26700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 >> v1:172.21.15.29:6789/0 conn(0x7f267013f710 legacy=0x7f26701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26700b44c0 con 0x7f26701430c0 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f26680019f0 con 0x7f26701430c0 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f2676197640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f2676197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f266e7fc640 10 monclient: dump: 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.122+0000 7f2676197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2668001b50 con 0x7f26701430c0 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 1 -- 172.21.15.29:0/4091435448 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2668001f20 con 0x7f26701430c0 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f2676197640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f2676197640 4 set_mon_vals no callback set 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.137+0000 7f266e7fc640 10 monclient: dump: 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 10 monclient: shutdown 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 1 -- 172.21.15.29:0/4091435448 >> v1:172.21.15.29:6790/0 conn(0x7f26701430c0 legacy=0x7f267014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 1 -- 172.21.15.29:0/4091435448 shutdown_connections 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 1 -- 172.21.15.29:0/4091435448 >> 172.21.15.29:0/4091435448 conn(0x7f26700b7ea0 msgr2=0x7f26700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 1 -- 172.21.15.29:0/4091435448 shutdown_connections 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.138+0000 7f2676197640 1 -- 172.21.15.29:0/4091435448 wait complete. 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.139+0000 7f2676197640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.139+0000 7f2676197640 10 monclient: monmap: 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.139+0000 7f2676197640 1 librados: starting msgr at 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.139+0000 7f2676197640 1 librados: starting objecter 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.139+0000 7f2676197640 1 Processor -- start 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 -- start start 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 librados: setting wanted keys 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 librados: calling monclient init 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient: init 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient(hunting): picked mon.noname-b con 0x7f26700a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient(hunting): picked mon.noname-c con 0x7f267013f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient(hunting): picked mon.noname-a con 0x7f26701430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26700ab920 con 0x7f26701430c0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26701db740 con 0x7f26700a98f0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26701db380 con 0x7f267013f710 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.140+0000 7f2676197640 10 monclient(hunting): authenticate will time out at 6164.893840s 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.145+0000 7f266effd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f267013f710 0x7f26700b0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:43662/0 (socket says 172.21.15.29:43662) 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.145+0000 7f266effd640 1 -- 172.21.15.29:0/3419095228 learned_addr learned my addr 172.21.15.29:0/3419095228 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:48.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.148+0000 7f2657fff640 10 client.?.objecter ms_handle_connect 0x7f267013f710 2024-03-25T19:34:48.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.148+0000 7f2657fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.157+0000 7f2657fff640 10 client.?.objecter ms_handle_connect 0x7f26701430c0 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.157+0000 7f2657fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.158+0000 7f2657fff640 10 client.?.objecter ms_handle_connect 0x7f26700a98f0 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.158+0000 7f2657fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275412779 0 0) 0x7f26701db380 con 0x7f267013f710 2024-03-25T19:34:48.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:34:48.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.167+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c004eb0 con 0x7f267013f710 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145868039 0 0) 0x7f265c004eb0 con 0x7f267013f710 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26701db380 con 0x7f267013f710 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2664001a30 con 0x7f267013f710 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 10 monclient(hunting): dump: 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:48.180 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.180+0000 7f2657fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400835404 0 0) 0x7f26701db740 con 0x7f26700a98f0 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:34:48.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.181+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c004eb0 con 0x7f26700a98f0 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323014882 0 0) 0x7f26700ab920 con 0x7f26701430c0 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:34:48.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.188+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26701db740 con 0x7f26701430c0 2024-03-25T19:34:48.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767391922 0 0) 0x7f26701db380 con 0x7f267013f710 2024-03-25T19:34:48.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 >> v1:172.21.15.196:6789/0 conn(0x7f26700a98f0 legacy=0x7f26701dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 >> v1:172.21.15.29:6789/0 conn(0x7f26701430c0 legacy=0x7f2670137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 1 monclient: found mon.c 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26701db510 con 0x7f267013f710 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 5 monclient: authenticate success, global_id 4262 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 10 monclient: _renew_subs 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26700b0d50 con 0x7f267013f710 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 10 monclient: _renew_subs 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26700b0750 con 0x7f267013f710 2024-03-25T19:34:48.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.207+0000 7f2676197640 1 librados: init done 2024-03-25T19:34:48.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.208+0000 7f2676197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.208+0000 7f2676197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.208+0000 7f2676197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.208+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26700627b0 con 0x7f267013f710 2024-03-25T19:34:48.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2664001cb0 con 0x7f267013f710 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2664003a60 con 0x7f267013f710 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f2657fff640 10 monclient: dump: 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:48.518 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:48.519 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:48.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.518+0000 7f266dffb640 4 set_mon_vals no callback set 2024-03-25T19:34:48.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.519+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f2664003c80 con 0x7f267013f710 2024-03-25T19:34:48.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.519+0000 7f266f7fe640 10 monclient: get_auth_request con 0x7f265c06de80 auth_method 0 2024-03-25T19:34:48.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.520+0000 7f2657fff640 10 client.4262.objecter ms_handle_connect 0x7f265c06de80 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f26640548c0 con 0x7f267013f710 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 10 client.4262.objecter ms_dispatch 0x7f2670000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26640503d0 con 0x7f267013f710 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.524+0000 7f2657fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.758+0000 7f2676197640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.758+0000 7f2676197640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.758+0000 7f2676197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.758+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f267013bfb0 con 0x7f267013f710 2024-03-25T19:34:48.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.819+0000 7f2657fff640 1 -- 172.21.15.29:0/3419095228 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f2664052300 con 0x7f267013f710 2024-03-25T19:34:48.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.819+0000 7f2657fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.819+0000 7f2657fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:48.819 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:48.819 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 10 librados: watch_flush enter 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 10 librados: watch_flush exit 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 10 client.4262.objecter successfully canceled tick 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 >> v1:172.21.15.29:6800/2314944989 conn(0x7f265c06de80 legacy=0x7f265c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 10 monclient: shutdown 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 >> v1:172.21.15.29:6790/0 conn(0x7f267013f710 legacy=0x7f26700b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 shutdown_connections 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 >> 172.21.15.29:0/3419095228 conn(0x7f26700b7ea0 msgr2=0x7f2670074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 shutdown_connections 2024-03-25T19:34:48.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 -- 172.21.15.29:0/3419095228 wait complete. 2024-03-25T19:34:48.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:48.822+0000 7f2676197640 1 librados: shutdown 2024-03-25T19:34:48.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:55.639 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.752+0000 7f14996cf640 10 monclient: get_monmap_and_config 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.752+0000 7f14996cf640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.752+0000 7f14996cf640 10 monclient: monmap: 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:34:55.753729+0000 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:34:55.753729+0000 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.753+0000 7f14996cf640 1 Processor -- start 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 1 -- start start 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient: init 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-a con 0x7f14940a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-c con 0x7f149413f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-b con 0x7f14941430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1494142520 con 0x7f14940a98f0 2024-03-25T19:34:55.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14940b28a0 con 0x7f149413f710 2024-03-25T19:34:55.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14940b2a30 con 0x7f14941430c0 2024-03-25T19:34:55.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.754+0000 7f14996cf640 10 monclient(hunting): authenticate will time out at 6172.507586s 2024-03-25T19:34:55.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.762+0000 7f148bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f149413f710 0x7f14941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:58160/0 (socket says 172.21.15.29:58160) 2024-03-25T19:34:55.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.762+0000 7f148bfff640 1 -- 172.21.15.29:0/3679199163 learned_addr learned my addr 172.21.15.29:0/3679199163 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717020231 0 0) 0x7f14940b28a0 con 0x7f149413f710 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:34:55.779 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.779+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1480003520 con 0x7f149413f710 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.784+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164189432 0 0) 0x7f14940b2a30 con 0x7f14941430c0 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.784+0000 7f14927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.784+0000 7f14927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.784+0000 7f14927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.784+0000 7f14927fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:34:55.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.785+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14940b28a0 con 0x7f14941430c0 2024-03-25T19:34:55.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186551207 0 0) 0x7f1480003520 con 0x7f149413f710 2024-03-25T19:34:55.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 >> v1:172.21.15.29:6790/0 conn(0x7f14941430c0 legacy=0x7f149414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 >> v1:172.21.15.29:6789/0 conn(0x7f14940a98f0 legacy=0x7f14940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 monclient: found mon.noname-c 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14940b44c0 con 0x7f149413f710 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f147c001690 con 0x7f149413f710 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14927fc640 10 monclient: dump: 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.796 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14996cf640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:34:55.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.796+0000 7f14996cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:55.804 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.804+0000 7f14927fc640 1 -- 172.21.15.29:0/3679199163 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f147c0017f0 con 0x7f149413f710 2024-03-25T19:34:55.804 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.804+0000 7f14927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:55.804 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.804+0000 7f14996cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:55.804 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.804+0000 7f14996cf640 4 set_mon_vals no callback set 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 10 monclient: shutdown 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 1 -- 172.21.15.29:0/3679199163 >> v1:172.21.15.196:6789/0 conn(0x7f149413f710 legacy=0x7f14941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 1 -- 172.21.15.29:0/3679199163 shutdown_connections 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 1 -- 172.21.15.29:0/3679199163 >> 172.21.15.29:0/3679199163 conn(0x7f14940b7ea0 msgr2=0x7f14940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 1 -- 172.21.15.29:0/3679199163 shutdown_connections 2024-03-25T19:34:55.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.805+0000 7f14996cf640 1 -- 172.21.15.29:0/3679199163 wait complete. 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.806+0000 7f14996cf640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.806+0000 7f14996cf640 10 monclient: monmap: 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.806+0000 7f14996cf640 1 librados: starting msgr at 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.806+0000 7f14996cf640 1 librados: starting objecter 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 Processor -- start 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 -- start start 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 librados: setting wanted keys 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 librados: calling monclient init 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient: init 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-b con 0x7f14940a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-a con 0x7f149413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient(hunting): picked mon.noname-c con 0x7f14941430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1494137cd0 con 0x7f149413f710 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1494136e50 con 0x7f14940a98f0 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1494137130 con 0x7f14941430c0 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.807+0000 7f14996cf640 10 monclient(hunting): authenticate will time out at 6172.560738s 2024-03-25T19:34:55.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.822+0000 7f148bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f149413f710 0x7f14941df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37690/0 (socket says 172.21.15.29:37690) 2024-03-25T19:34:55.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.822+0000 7f148bfff640 1 -- 172.21.15.29:0/1527925181 learned_addr learned my addr 172.21.15.29:0/1527925181 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:34:55.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.823+0000 7f148b7fe640 10 client.?.objecter ms_handle_connect 0x7f149413f710 2024-03-25T19:34:55.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.823+0000 7f148b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.825+0000 7f148b7fe640 10 client.?.objecter ms_handle_connect 0x7f14941430c0 2024-03-25T19:34:55.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.825+0000 7f148b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.831+0000 7f148b7fe640 10 client.?.objecter ms_handle_connect 0x7f14940a98f0 2024-03-25T19:34:55.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.831+0000 7f148b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.837+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248593904 0 0) 0x7f1494137130 con 0x7f14941430c0 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.838+0000 7f148b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.838+0000 7f148b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.838+0000 7f148b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.838+0000 7f148b7fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:34:55.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.838+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474004eb0 con 0x7f14941430c0 2024-03-25T19:34:55.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271523049 0 0) 0x7f1494137cd0 con 0x7f149413f710 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.842+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1494137130 con 0x7f149413f710 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698778256 0 0) 0x7f1494136e50 con 0x7f14940a98f0 2024-03-25T19:34:55.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:34:55.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.843+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1494137cd0 con 0x7f14940a98f0 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417705894 0 0) 0x7f1474004eb0 con 0x7f14941430c0 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1494136e50 con 0x7f14941430c0 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f148c001bd0 con 0x7f14941430c0 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:55.848 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:55.849 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.848+0000 7f148b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563106254 0 0) 0x7f1494137cd0 con 0x7f14940a98f0 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1474004eb0 con 0x7f14940a98f0 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1484001860 con 0x7f14940a98f0 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.862+0000 7f148b7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:55.862 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:55.863 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:55.863 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:55.863 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304698609 0 0) 0x7f1494137130 con 0x7f149413f710 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1494137cd0 con 0x7f149413f710 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f147c001ce0 con 0x7f149413f710 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.865+0000 7f148b7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:55.865 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:55.866 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2770361562 0 0) 0x7f1494136e50 con 0x7f14941430c0 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 >> v1:172.21.15.196:6789/0 conn(0x7f14940a98f0 legacy=0x7f14941375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 >> v1:172.21.15.29:6789/0 conn(0x7f149413f710 legacy=0x7f14941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 1 monclient: found mon.c 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:55.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14940b78e0 con 0x7f14941430c0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f148b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 5 monclient: authenticate success, global_id 4268 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 10 monclient: _renew_subs 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14940b7bc0 con 0x7f14941430c0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 10 monclient: _renew_subs 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1494135300 con 0x7f14941430c0 2024-03-25T19:34:55.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.867+0000 7f14996cf640 1 librados: init done 2024-03-25T19:34:55.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.868+0000 7f14996cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.868+0000 7f14996cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.868+0000 7f14996cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:55.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:55.868+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14940627b0 con 0x7f14941430c0 2024-03-25T19:34:56.196 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f148c0020c0 con 0x7f14941430c0 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f148c003b10 con 0x7f14941430c0 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f148b7fe640 10 monclient: dump: 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.197 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.197+0000 7f1491ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:56.198 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.198+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f148c019ba0 con 0x7f14941430c0 2024-03-25T19:34:56.199 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.198+0000 7f1492ffd640 10 monclient: get_auth_request con 0x7f147406e0f0 auth_method 0 2024-03-25T19:34:56.199 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.199+0000 7f148b7fe640 10 client.4268.objecter ms_handle_connect 0x7f147406e0f0 2024-03-25T19:34:56.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f148c053880 con 0x7f14941430c0 2024-03-25T19:34:56.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 10 client.4268.objecter ms_dispatch 0x7f1494000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:56.203 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:56.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:56.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T19:34:56.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f148c04f1c0 con 0x7f14941430c0 2024-03-25T19:34:56.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:56.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.204+0000 7f148b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:56.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.425+0000 7f14996cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.425+0000 7f14996cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.425+0000 7f14996cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:34:56.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.425+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f149413e230 con 0x7f14941430c0 2024-03-25T19:34:56.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.456+0000 7f148b7fe640 1 -- 172.21.15.29:0/1527925181 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f148c0512c0 con 0x7f14941430c0 2024-03-25T19:34:56.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.456+0000 7f148b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.456+0000 7f148b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:56.456 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:34:56.456 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 10 librados: watch_flush enter 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 10 librados: watch_flush exit 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 10 client.4268.objecter successfully canceled tick 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 >> v1:172.21.15.29:6800/2314944989 conn(0x7f147406e0f0 legacy=0x7f14740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 10 monclient: shutdown 2024-03-25T19:34:56.458 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 >> v1:172.21.15.29:6790/0 conn(0x7f14941430c0 legacy=0x7f14941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 shutdown_connections 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 >> 172.21.15.29:0/1527925181 conn(0x7f14940b7ea0 msgr2=0x7f1494074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 shutdown_connections 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 -- 172.21.15.29:0/1527925181 wait complete. 2024-03-25T19:34:56.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:34:56.458+0000 7f14996cf640 1 librados: shutdown 2024-03-25T19:34:56.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:03.275 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:03.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.394+0000 7f68d2064640 10 monclient: get_monmap_and_config 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.394+0000 7f68d2064640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.394+0000 7f68d2064640 10 monclient: monmap: 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:03.395015+0000 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:03.395015+0000 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.395+0000 7f68d2064640 1 Processor -- start 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.395+0000 7f68d2064640 1 -- start start 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.395+0000 7f68d2064640 10 monclient: init 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:03.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:03.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc142520 con 0x7f68cc1430c0 2024-03-25T19:35:03.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0b28a0 con 0x7f68cc13f710 2024-03-25T19:35:03.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0b2a30 con 0x7f68cc0a98f0 2024-03-25T19:35:03.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.396+0000 7f68d2064640 10 monclient(hunting): authenticate will time out at 6180.149277s 2024-03-25T19:35:03.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.407+0000 7f68cbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f68cc1430c0 0x7f68cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40074/0 (socket says 172.21.15.29:40074) 2024-03-25T19:35:03.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.407+0000 7f68cbfff640 1 -- 172.21.15.29:0/1184164851 learned_addr learned my addr 172.21.15.29:0/1184164851 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:03.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534201140 0 0) 0x7f68cc142520 con 0x7f68cc1430c0 2024-03-25T19:35:03.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:35:03.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.418+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b4003520 con 0x7f68cc1430c0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653108663 0 0) 0x7f68b4003520 con 0x7f68cc1430c0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 >> v1:172.21.15.29:6790/0 conn(0x7f68cc0a98f0 legacy=0x7f68cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 >> v1:172.21.15.196:6789/0 conn(0x7f68cc13f710 legacy=0x7f68cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc0b44c0 con 0x7f68cc1430c0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f68c4001a10 con 0x7f68cc1430c0 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68ca7fc640 10 monclient: dump: 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68d2064640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:35:03.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.423+0000 7f68d2064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c4001b70 con 0x7f68cc1430c0 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 1 -- 172.21.15.29:0/1184164851 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f68c4001fc0 con 0x7f68cc1430c0 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68ca7fc640 10 monclient: dump: 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.444 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68d2064640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:03.445 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.445+0000 7f68d2064640 4 set_mon_vals no callback set 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 10 monclient: shutdown 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 1 -- 172.21.15.29:0/1184164851 >> v1:172.21.15.29:6789/0 conn(0x7f68cc1430c0 legacy=0x7f68cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 1 -- 172.21.15.29:0/1184164851 shutdown_connections 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 1 -- 172.21.15.29:0/1184164851 >> 172.21.15.29:0/1184164851 conn(0x7f68cc0b7ea0 msgr2=0x7f68cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 1 -- 172.21.15.29:0/1184164851 shutdown_connections 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.446+0000 7f68d2064640 1 -- 172.21.15.29:0/1184164851 wait complete. 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 10 monclient: monmap: 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:03.447 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 librados: starting msgr at 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 librados: starting objecter 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 Processor -- start 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 -- start start 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 librados: setting wanted keys 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 1 librados: calling monclient init 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.447+0000 7f68d2064640 10 monclient: init 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0abac0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0abda0 con 0x7f68cc13f710 2024-03-25T19:35:03.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc1380e0 con 0x7f68cc1430c0 2024-03-25T19:35:03.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.448+0000 7f68d2064640 10 monclient(hunting): authenticate will time out at 6180.201859s 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.457+0000 7f68caffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f68cc13f710 0x7f68cc138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:40624/0 (socket says 172.21.15.29:40624) 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.457+0000 7f68caffd640 1 -- 172.21.15.29:0/1101568617 learned_addr learned my addr 172.21.15.29:0/1101568617 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.458+0000 7f68b3fff640 10 client.?.objecter ms_handle_connect 0x7f68cc13f710 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.458+0000 7f68b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.463+0000 7f68b3fff640 10 client.?.objecter ms_handle_connect 0x7f68cc0a98f0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.463+0000 7f68b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.464+0000 7f68b3fff640 10 client.?.objecter ms_handle_connect 0x7f68cc1430c0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.464+0000 7f68b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271221385 0 0) 0x7f68cc0abac0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.471+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b8004eb0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044895535 0 0) 0x7f68cc0abda0 con 0x7f68cc13f710 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.473+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68cc0abac0 con 0x7f68cc13f710 2024-03-25T19:35:03.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541274148 0 0) 0x7f68cc0abac0 con 0x7f68cc13f710 2024-03-25T19:35:03.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68cc0abda0 con 0x7f68cc13f710 2024-03-25T19:35:03.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f68c0001970 con 0x7f68cc13f710 2024-03-25T19:35:03.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 10 monclient(hunting): dump: 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.482+0000 7f68b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 517858868 0 0) 0x7f68b8004eb0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68cc0abac0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f68bc001930 con 0x7f68cc0a98f0 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.483+0000 7f68b3fff640 10 monclient(hunting): dump: 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478988478 0 0) 0x7f68cc1380e0 con 0x7f68cc1430c0 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:35:03.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.486+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b8004eb0 con 0x7f68cc1430c0 2024-03-25T19:35:03.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3256533948 0 0) 0x7f68cc0abac0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 >> v1:172.21.15.29:6790/0 conn(0x7f68cc1430c0 legacy=0x7f68cc1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 >> v1:172.21.15.196:6789/0 conn(0x7f68cc13f710 legacy=0x7f68cc138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 1 monclient: found mon.a 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc137d90 con 0x7f68cc0a98f0 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 10 monclient: _renew_subs 2024-03-25T19:35:03.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc139240 con 0x7f68cc0a98f0 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 10 monclient: _renew_subs 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc136830 con 0x7f68cc0a98f0 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.495+0000 7f68d2064640 1 librados: init done 2024-03-25T19:35:03.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.496+0000 7f68d2064640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.496+0000 7f68d2064640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.496+0000 7f68d2064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:03.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.496+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc0627b0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68bc0035f0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68c9ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f68bc003980 con 0x7f68cc0a98f0 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.818+0000 7f68b3fff640 10 monclient: dump: 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:03.818 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:03.819 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:03.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.819+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f68bc003ba0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.819+0000 7f68caffd640 10 monclient: get_auth_request con 0x7f68b806df40 auth_method 0 2024-03-25T19:35:03.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.820+0000 7f68b3fff640 10 client.4272.objecter ms_handle_connect 0x7f68b806df40 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f68bc0536c0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 10 client.4272.objecter ms_dispatch 0x7f68cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68bc04f1d0 con 0x7f68cc0a98f0 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:03.829+0000 7f68b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:04.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.042+0000 7f68d2064640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.042+0000 7f68d2064640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.042+0000 7f68d2064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:04.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.042+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68cc13f070 con 0x7f68cc0a98f0 2024-03-25T19:35:04.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.066+0000 7f68b3fff640 1 -- 172.21.15.29:0/1101568617 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f68bc051100 con 0x7f68cc0a98f0 2024-03-25T19:35:04.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.066+0000 7f68b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.066+0000 7f68b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:04.065 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:04.065 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 10 librados: watch_flush enter 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 10 librados: watch_flush exit 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 10 client.4272.objecter successfully canceled tick 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 >> v1:172.21.15.29:6800/2314944989 conn(0x7f68b806df40 legacy=0x7f68b8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 10 monclient: shutdown 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 >> v1:172.21.15.29:6789/0 conn(0x7f68cc0a98f0 legacy=0x7f68cc0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 shutdown_connections 2024-03-25T19:35:04.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.068+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 >> 172.21.15.29:0/1101568617 conn(0x7f68cc0b7ea0 msgr2=0x7f68cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:04.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.069+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 shutdown_connections 2024-03-25T19:35:04.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.069+0000 7f68d2064640 1 -- 172.21.15.29:0/1101568617 wait complete. 2024-03-25T19:35:04.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:04.069+0000 7f68d2064640 1 librados: shutdown 2024-03-25T19:35:04.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:04.323 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:34.487 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.374 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:34.538 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.416 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:34.580 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.426 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:34.588 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.441 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:34.603 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.454 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:34.618 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.464 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:34.628 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.489 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:34.652 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:04.512 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:34.676 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:10.885 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.001+0000 7fa0fa313640 10 monclient: get_monmap_and_config 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.001+0000 7fa0fa313640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.001+0000 7fa0fa313640 10 monclient: monmap: 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:11.002442+0000 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:11.002442+0000 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:11.001 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:11.002 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:11.002 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.002+0000 7fa0fa313640 1 Processor -- start 2024-03-25T19:35:11.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.002+0000 7fa0fa313640 1 -- start start 2024-03-25T19:35:11.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.002+0000 7fa0fa313640 10 monclient: init 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f40a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f413f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f41430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f4142520 con 0x7fa0f40a98f0 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40b28a0 con 0x7fa0f413f710 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40b2a30 con 0x7fa0f41430c0 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.003+0000 7fa0fa313640 10 monclient(hunting): authenticate will time out at 6187.756757s 2024-03-25T19:35:11.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.010+0000 7fa0f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa0f40a98f0 0x7fa0f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34966/0 (socket says 172.21.15.29:34966) 2024-03-25T19:35:11.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.010+0000 7fa0f3fff640 1 -- 172.21.15.29:0/2472257440 learned_addr learned my addr 172.21.15.29:0/2472257440 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732704775 0 0) 0x7fa0f4142520 con 0x7fa0f40a98f0 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:35:11.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.024+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc003520 con 0x7fa0f40a98f0 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.025+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888645853 0 0) 0x7fa0f40b2a30 con 0x7fa0f41430c0 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.026+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.026+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.026+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.026+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:35:11.025 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.026+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f4142520 con 0x7fa0f41430c0 2024-03-25T19:35:11.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735169912 0 0) 0x7fa0f40b28a0 con 0x7fa0f413f710 2024-03-25T19:35:11.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:35:11.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.031+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f40b2a30 con 0x7fa0f413f710 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720870956 0 0) 0x7fa0f4142520 con 0x7fa0f41430c0 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 >> v1:172.21.15.196:6789/0 conn(0x7fa0f413f710 legacy=0x7fa0f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 >> v1:172.21.15.29:6789/0 conn(0x7fa0f40a98f0 legacy=0x7fa0f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:35:11.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40b44c0 con 0x7fa0f41430c0 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.034+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa0ec0019f0 con 0x7fa0f41430c0 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.035+0000 7fa0f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.035+0000 7fa0f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.035+0000 7fa0f2ffd640 10 monclient: dump: 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.035 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.035+0000 7fa0fa313640 5 monclient: authenticate success, global_id 4277 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.035+0000 7fa0fa313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:11.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec001b50 con 0x7fa0f41430c0 2024-03-25T19:35:11.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 1 -- 172.21.15.29:0/2472257440 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa0ec001f60 con 0x7fa0f41430c0 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0fa313640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0fa313640 4 set_mon_vals no callback set 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.057+0000 7fa0f2ffd640 10 monclient: dump: 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 10 monclient: shutdown 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 1 -- 172.21.15.29:0/2472257440 >> v1:172.21.15.29:6790/0 conn(0x7fa0f41430c0 legacy=0x7fa0f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 1 -- 172.21.15.29:0/2472257440 shutdown_connections 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 1 -- 172.21.15.29:0/2472257440 >> 172.21.15.29:0/2472257440 conn(0x7fa0f40b7ea0 msgr2=0x7fa0f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 1 -- 172.21.15.29:0/2472257440 shutdown_connections 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.058+0000 7fa0fa313640 1 -- 172.21.15.29:0/2472257440 wait complete. 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.059+0000 7fa0fa313640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.059+0000 7fa0fa313640 10 monclient: monmap: 2024-03-25T19:35:11.060 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.059+0000 7fa0fa313640 1 librados: starting msgr at 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.059+0000 7fa0fa313640 1 librados: starting objecter 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.059+0000 7fa0fa313640 1 Processor -- start 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 -- start start 2024-03-25T19:35:11.061 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 librados: setting wanted keys 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 librados: calling monclient init 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient: init 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f40a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f41430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f4138600 con 0x7fa0f413f710 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40b1620 con 0x7fa0f41430c0 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f41346b0 con 0x7fa0f40a98f0 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.060+0000 7fa0fa313640 10 monclient(hunting): authenticate will time out at 6187.813627s 2024-03-25T19:35:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.076+0000 7fa0f88b0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa0f41430c0 0x7fa0f41da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:39460/0 (socket says 172.21.15.29:39460) 2024-03-25T19:35:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.076+0000 7fa0f88b0640 1 -- 172.21.15.29:0/2027685855 learned_addr learned my addr 172.21.15.29:0/2027685855 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.077+0000 7fa0f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0f41430c0 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.077+0000 7fa0f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528592718 0 0) 0x7fa0f40b1620 con 0x7fa0f41430c0 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:35:11.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.083+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0004eb0 con 0x7fa0f41430c0 2024-03-25T19:35:11.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182548504 0 0) 0x7fa0e0004eb0 con 0x7fa0f41430c0 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f40b1620 con 0x7fa0f41430c0 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa0ec001fa0 con 0x7fa0f41430c0 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0f40a98f0 2024-03-25T19:35:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa0f413f710 2024-03-25T19:35:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.092+0000 7fa0f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239789193 0 0) 0x7fa0f41346b0 con 0x7fa0f40a98f0 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:35:11.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.101+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0004eb0 con 0x7fa0f40a98f0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086768892 0 0) 0x7fa0f40b1620 con 0x7fa0f41430c0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 >> v1:172.21.15.29:6790/0 conn(0x7fa0f40a98f0 legacy=0x7fa0f4137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 >> v1:172.21.15.29:6789/0 conn(0x7fa0f413f710 legacy=0x7fa0f4133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 1 monclient: found mon.b 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4134840 con 0x7fa0f41430c0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 10 monclient: _renew_subs 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f4137f50 con 0x7fa0f41430c0 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 10 monclient: _renew_subs 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f41daf90 con 0x7fa0f41430c0 2024-03-25T19:35:11.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.103+0000 7fa0fa313640 1 librados: init done 2024-03-25T19:35:11.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.104+0000 7fa0fa313640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.104+0000 7fa0fa313640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.104+0000 7fa0fa313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:11.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.104+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40627b0 con 0x7fa0f41430c0 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec003c20 con 0x7fa0f41430c0 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa0ec002100 con 0x7fa0f41430c0 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f0ff9640 10 monclient: dump: 2024-03-25T19:35:11.382 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.382+0000 7fa0f27fc640 4 set_mon_vals no callback set 2024-03-25T19:35:11.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.383+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fa0ec0040f0 con 0x7fa0f41430c0 2024-03-25T19:35:11.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.383+0000 7fa0f3fff640 10 monclient: get_auth_request con 0x7fa0e006ddd0 auth_method 0 2024-03-25T19:35:11.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.383+0000 7fa0f0ff9640 10 client.4276.objecter ms_handle_connect 0x7fa0e006ddd0 2024-03-25T19:35:11.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fa0ec054c10 con 0x7fa0f41430c0 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 10 client.4276.objecter ms_dispatch 0x7fa0f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0ec050720 con 0x7fa0f41430c0 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.387+0000 7fa0f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:11.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.617+0000 7fa0fa313640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.617 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.617+0000 7fa0fa313640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.617 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.617+0000 7fa0fa313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:11.617 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.617+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f41387b0 con 0x7fa0f41430c0 2024-03-25T19:35:11.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.652+0000 7fa0f0ff9640 1 -- 172.21.15.29:0/2027685855 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fa0ec052650 con 0x7fa0f41430c0 2024-03-25T19:35:11.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.652+0000 7fa0f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.652+0000 7fa0f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:11.652 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:11.652 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 10 librados: watch_flush enter 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 10 librados: watch_flush exit 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 >> v1:172.21.15.29:6800/2314944989 conn(0x7fa0e006ddd0 legacy=0x7fa0e0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 10 monclient: shutdown 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 >> v1:172.21.15.196:6789/0 conn(0x7fa0f41430c0 legacy=0x7fa0f41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 shutdown_connections 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 >> 172.21.15.29:0/2027685855 conn(0x7fa0f40b7ea0 msgr2=0x7fa0f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 shutdown_connections 2024-03-25T19:35:11.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 -- 172.21.15.29:0/2027685855 wait complete. 2024-03-25T19:35:11.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:11.655+0000 7fa0fa313640 1 librados: shutdown 2024-03-25T19:35:11.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:18.473 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:18.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 10 monclient: get_monmap_and_config 2024-03-25T19:35:18.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 10 monclient: monmap: 2024-03-25T19:35:18.587 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:18.587 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:18.588901+0000 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:18.588901+0000 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:18.588 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 1 Processor -- start 2024-03-25T19:35:18.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 1 -- start start 2024-03-25T19:35:18.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.587+0000 7f432a2f1640 10 monclient: init 2024-03-25T19:35:18.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4324061830 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f432406ff10 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4324072ae0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4324072f90 con 0x7f4324061830 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4324070bb0 con 0x7f432406ff10 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43240719f0 con 0x7f4324072ae0 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.588+0000 7f432a2f1640 10 monclient(hunting): authenticate will time out at 6195.342620s 2024-03-25T19:35:18.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.601+0000 7f43237fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f432406ff10 0x7f432413bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50920/0 (socket says 172.21.15.29:50920) 2024-03-25T19:35:18.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.601+0000 7f43237fe640 1 -- 172.21.15.29:0/3180558647 learned_addr learned my addr 172.21.15.29:0/3180558647 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:18.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422821222 0 0) 0x7f43240719f0 con 0x7f4324072ae0 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.614+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f430c005730 con 0x7f4324072ae0 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224059815 0 0) 0x7f4324070bb0 con 0x7f432406ff10 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:35:18.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.615+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43240719f0 con 0x7f432406ff10 2024-03-25T19:35:18.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2617270975 0 0) 0x7f43240719f0 con 0x7f432406ff10 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 >> v1:172.21.15.29:6790/0 conn(0x7f4324072ae0 legacy=0x7f432413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 >> v1:172.21.15.29:6789/0 conn(0x7f4324061830 legacy=0x7f43240723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 monclient: found mon.noname-c 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324071cd0 con 0x7f432406ff10 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4318001670 con 0x7f432406ff10 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:18.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f4322ffd640 10 monclient: dump: 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f432a2f1640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:35:18.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.626+0000 7f432a2f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:18.632 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43180017d0 con 0x7f432406ff10 2024-03-25T19:35:18.632 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.632 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 1 -- 172.21.15.29:0/3180558647 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4318001b60 con 0x7f432406ff10 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f432a2f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f432a2f1640 4 set_mon_vals no callback set 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.632+0000 7f4322ffd640 10 monclient: dump: 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:18.633 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:18.634 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:18.634 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 10 monclient: shutdown 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 1 -- 172.21.15.29:0/3180558647 >> v1:172.21.15.196:6789/0 conn(0x7f432406ff10 legacy=0x7f432413bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 1 -- 172.21.15.29:0/3180558647 shutdown_connections 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 1 -- 172.21.15.29:0/3180558647 >> 172.21.15.29:0/3180558647 conn(0x7f43240b11e0 msgr2=0x7f43240b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 1 -- 172.21.15.29:0/3180558647 shutdown_connections 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 1 -- 172.21.15.29:0/3180558647 wait complete. 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.633+0000 7f432a2f1640 10 monclient: monmap: 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 librados: starting msgr at 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 librados: starting objecter 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 Processor -- start 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 -- start start 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 librados: setting wanted keys 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 1 librados: calling monclient init 2024-03-25T19:35:18.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.634+0000 7f432a2f1640 10 monclient: init 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4324061830 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4324072ae0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f43240b2890 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43241d30c0 con 0x7f4324061830 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43240b3570 con 0x7f4324072ae0 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43241d2c50 con 0x7f43240b2890 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.635+0000 7f432a2f1640 10 monclient(hunting): authenticate will time out at 6195.389656s 2024-03-25T19:35:18.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.638+0000 7f43237fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4324072ae0 0x7f43240b2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50942/0 (socket says 172.21.15.29:50942) 2024-03-25T19:35:18.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.638+0000 7f43237fe640 1 -- 172.21.15.29:0/2586027012 learned_addr learned my addr 172.21.15.29:0/2586027012 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:18.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.640+0000 7f4320ff9640 10 client.?.objecter ms_handle_connect 0x7f4324072ae0 2024-03-25T19:35:18.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.640+0000 7f4320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587237442 0 0) 0x7f43240b3570 con 0x7f4324072ae0 2024-03-25T19:35:18.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:35:18.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.646+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4310004eb0 con 0x7f4324072ae0 2024-03-25T19:35:18.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.654+0000 7f4320ff9640 10 client.?.objecter ms_handle_connect 0x7f43240b2890 2024-03-25T19:35:18.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.654+0000 7f4320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.657+0000 7f4320ff9640 10 client.?.objecter ms_handle_connect 0x7f4324061830 2024-03-25T19:35:18.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.657+0000 7f4320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351167432 0 0) 0x7f4310004eb0 con 0x7f4324072ae0 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43240b3570 con 0x7f4324072ae0 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4318001ae0 con 0x7f4324072ae0 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.669 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:18.670 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:18.670 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:18.670 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:18.670 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.668+0000 7f4320ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:18.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536787621 0 0) 0x7f43241d30c0 con 0x7f4324061830 2024-03-25T19:35:18.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:35:18.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.672+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4310004eb0 con 0x7f4324061830 2024-03-25T19:35:18.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915726837 0 0) 0x7f43241d2c50 con 0x7f43240b2890 2024-03-25T19:35:18.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:35:18.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.674+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43241d30c0 con 0x7f43240b2890 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2981750129 0 0) 0x7f43240b3570 con 0x7f4324072ae0 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 >> v1:172.21.15.29:6790/0 conn(0x7f43240b2890 legacy=0x7f43241dd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 >> v1:172.21.15.29:6789/0 conn(0x7f4324061830 legacy=0x7f43241d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 1 monclient: found mon.b 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43241d2de0 con 0x7f4324072ae0 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f4320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 10 monclient: _renew_subs 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4324139b10 con 0x7f4324072ae0 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 10 monclient: _renew_subs 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43241def00 con 0x7f4324072ae0 2024-03-25T19:35:18.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.684+0000 7f432a2f1640 1 librados: init done 2024-03-25T19:35:18.687 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.685+0000 7f432a2f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.687 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.685+0000 7f432a2f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.687 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.685+0000 7f432a2f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:18.687 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.685+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43240627b0 con 0x7f4324072ae0 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4318004a90 con 0x7f4324072ae0 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4318004da0 con 0x7f4324072ae0 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f4320ff9640 10 monclient: dump: 2024-03-25T19:35:18.956 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:18.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.955+0000 7f43227fc640 4 set_mon_vals no callback set 2024-03-25T19:35:18.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.956+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f431801ad70 con 0x7f4324072ae0 2024-03-25T19:35:18.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.957+0000 7f4323fff640 10 monclient: get_auth_request con 0x7f431006de10 auth_method 0 2024-03-25T19:35:18.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.957+0000 7f4320ff9640 10 client.4282.objecter ms_handle_connect 0x7f431006de10 2024-03-25T19:35:18.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.962+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f4318054990 con 0x7f4324072ae0 2024-03-25T19:35:18.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.962+0000 7f4320ff9640 10 client.4282.objecter ms_dispatch 0x7f4324000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:18.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.962+0000 7f4320ff9640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:18.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.962+0000 7f4320ff9640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:18.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.962+0000 7f4320ff9640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:35:18.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.966+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43180504a0 con 0x7f4324072ae0 2024-03-25T19:35:18.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.966+0000 7f4320ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:18.966+0000 7f4320ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:19.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.218+0000 7f432a2f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.218+0000 7f432a2f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.218+0000 7f432a2f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:19.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.219+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43241d3270 con 0x7f4324072ae0 2024-03-25T19:35:19.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.242+0000 7f4320ff9640 1 -- 172.21.15.29:0/2586027012 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f43241d3270 con 0x7f4324072ae0 2024-03-25T19:35:19.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.242+0000 7f4320ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.242+0000 7f4320ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:19.243 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:19.243 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 10 librados: watch_flush enter 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 10 librados: watch_flush exit 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 >> v1:172.21.15.29:6800/2314944989 conn(0x7f431006de10 legacy=0x7f43100702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 10 monclient: shutdown 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 >> v1:172.21.15.196:6789/0 conn(0x7f4324072ae0 legacy=0x7f43240b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 shutdown_connections 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 >> 172.21.15.29:0/2586027012 conn(0x7f43240b11e0 msgr2=0x7f4324074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 shutdown_connections 2024-03-25T19:35:19.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 -- 172.21.15.29:0/2586027012 wait complete. 2024-03-25T19:35:19.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:19.245+0000 7f432a2f1640 1 librados: shutdown 2024-03-25T19:35:19.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:24.421 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:54.585 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.474 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:54.638 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.510 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:54.674 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.530 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:54.694 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.534 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:54.698 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.539 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:54.701 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.554 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:54.717 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.571 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:54.735 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:24.578 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:01:54.741 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:26.062 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:26.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.175+0000 7fd83961e640 10 monclient: get_monmap_and_config 2024-03-25T19:35:26.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.175+0000 7fd83961e640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.175+0000 7fd83961e640 10 monclient: monmap: 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:26.177027+0000 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:26.177027+0000 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:26.176 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 1 Processor -- start 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 1 -- start start 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient: init 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8340a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-b con 0x7fd83413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-c con 0x7fd8341430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd834142520 con 0x7fd8340a98f0 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340b28a0 con 0x7fd8341430c0 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340b2a30 con 0x7fd83413f710 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.176+0000 7fd83961e640 10 monclient(hunting): authenticate will time out at 6202.930787s 2024-03-25T19:35:26.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.184+0000 7fd8337fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd8341430c0 0x7fd83414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:42828/0 (socket says 172.21.15.29:42828) 2024-03-25T19:35:26.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.184+0000 7fd8337fe640 1 -- 172.21.15.29:0/1834421115 learned_addr learned my addr 172.21.15.29:0/1834421115 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:26.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911987004 0 0) 0x7fd8340b28a0 con 0x7fd8341430c0 2024-03-25T19:35:26.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:35:26.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.200+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd820003520 con 0x7fd8341430c0 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.201+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121476742 0 0) 0x7fd8340b2a30 con 0x7fd83413f710 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.201+0000 7fd8327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.201+0000 7fd8327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.201+0000 7fd8327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.201+0000 7fd8327fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:35:26.202 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.202+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8340b28a0 con 0x7fd83413f710 2024-03-25T19:35:26.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573447589 0 0) 0x7fd820003520 con 0x7fd8341430c0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 >> v1:172.21.15.29:6790/0 conn(0x7fd83413f710 legacy=0x7fd8341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 >> v1:172.21.15.29:6789/0 conn(0x7fd8340a98f0 legacy=0x7fd8340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 monclient: found mon.noname-c 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340b44c0 con 0x7fd8341430c0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd82c001a10 con 0x7fd8341430c0 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd83961e640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd83961e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd8327fc640 10 monclient: dump: 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.214 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.212+0000 7fd83961e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:26.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd82c001b70 con 0x7fd8341430c0 2024-03-25T19:35:26.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 1 -- 172.21.15.29:0/1834421115 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd82c001f80 con 0x7fd8341430c0 2024-03-25T19:35:26.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd8327fc640 10 monclient: dump: 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd83961e640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:26.236 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.234+0000 7fd83961e640 4 set_mon_vals no callback set 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 10 monclient: shutdown 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 1 -- 172.21.15.29:0/1834421115 >> v1:172.21.15.196:6789/0 conn(0x7fd8341430c0 legacy=0x7fd83414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 1 -- 172.21.15.29:0/1834421115 shutdown_connections 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 1 -- 172.21.15.29:0/1834421115 >> 172.21.15.29:0/1834421115 conn(0x7fd8340b7ea0 msgr2=0x7fd8340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 1 -- 172.21.15.29:0/1834421115 shutdown_connections 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.235+0000 7fd83961e640 1 -- 172.21.15.29:0/1834421115 wait complete. 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 10 monclient: monmap: 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:26.238 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 librados: starting msgr at 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 librados: starting objecter 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 Processor -- start 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 -- start start 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 librados: setting wanted keys 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 1 librados: calling monclient init 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.236+0000 7fd83961e640 10 monclient: init 2024-03-25T19:35:26.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-c con 0x7fd8340a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-b con 0x7fd83413f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8341430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd834071f10 con 0x7fd8341430c0 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340708f0 con 0x7fd83413f710 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd834070bd0 con 0x7fd8340a98f0 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.237+0000 7fd83961e640 10 monclient(hunting): authenticate will time out at 6202.991405s 2024-03-25T19:35:26.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.249+0000 7fd832ffd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fd8340a98f0 0x7fd834071640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:60624/0 (socket says 172.21.15.29:60624) 2024-03-25T19:35:26.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.249+0000 7fd832ffd640 1 -- 172.21.15.29:0/776162455 learned_addr learned my addr 172.21.15.29:0/776162455 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:26.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.254+0000 7fd82b7fe640 10 client.?.objecter ms_handle_connect 0x7fd8340a98f0 2024-03-25T19:35:26.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.254+0000 7fd82b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.260+0000 7fd82b7fe640 10 client.?.objecter ms_handle_connect 0x7fd83413f710 2024-03-25T19:35:26.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.260+0000 7fd82b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787453972 0 0) 0x7fd834070bd0 con 0x7fd8340a98f0 2024-03-25T19:35:26.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:35:26.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.267+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd814004eb0 con 0x7fd8340a98f0 2024-03-25T19:35:26.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.272+0000 7fd82b7fe640 10 client.?.objecter ms_handle_connect 0x7fd8341430c0 2024-03-25T19:35:26.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.272+0000 7fd82b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853259279 0 0) 0x7fd8340708f0 con 0x7fd83413f710 2024-03-25T19:35:26.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:35:26.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.273+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd834070bd0 con 0x7fd83413f710 2024-03-25T19:35:26.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1803360535 0 0) 0x7fd814004eb0 con 0x7fd8340a98f0 2024-03-25T19:35:26.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8340708f0 con 0x7fd8340a98f0 2024-03-25T19:35:26.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd824001a30 con 0x7fd8340a98f0 2024-03-25T19:35:26.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.283+0000 7fd82b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:26.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.287+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857775653 0 0) 0x7fd834070bd0 con 0x7fd83413f710 2024-03-25T19:35:26.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.287+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd814004eb0 con 0x7fd83413f710 2024-03-25T19:35:26.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.287+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd81c0018b0 con 0x7fd83413f710 2024-03-25T19:35:26.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.288+0000 7fd82b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.288+0000 7fd82b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.288+0000 7fd82b7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:26.289 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971784730 0 0) 0x7fd834071f10 con 0x7fd8341430c0 2024-03-25T19:35:26.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:35:26.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.290+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd834070bd0 con 0x7fd8341430c0 2024-03-25T19:35:26.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963410130 0 0) 0x7fd814004eb0 con 0x7fd83413f710 2024-03-25T19:35:26.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 >> v1:172.21.15.29:6790/0 conn(0x7fd8340a98f0 legacy=0x7fd834071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 >> v1:172.21.15.29:6789/0 conn(0x7fd8341430c0 legacy=0x7fd83406efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 1 monclient: found mon.b 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834138d30 con 0x7fd83413f710 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.308+0000 7fd82b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 10 monclient: _renew_subs 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834138fe0 con 0x7fd83413f710 2024-03-25T19:35:26.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 10 monclient: _renew_subs 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8341e7e90 con 0x7fd83413f710 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.309+0000 7fd83961e640 1 librados: init done 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.310+0000 7fd83961e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.310+0000 7fd83961e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.310+0000 7fd83961e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.310+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340627b0 con 0x7fd83413f710 2024-03-25T19:35:26.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd81c003610 con 0x7fd83413f710 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd81c003920 con 0x7fd83413f710 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd82b7fe640 10 monclient: dump: 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:26.597 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.596+0000 7fd831ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.597+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fd81c0198f0 con 0x7fd83413f710 2024-03-25T19:35:26.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.597+0000 7fd832ffd640 10 monclient: get_auth_request con 0x7fd81406dee0 auth_method 0 2024-03-25T19:35:26.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.598+0000 7fd82b7fe640 10 client.4288.objecter ms_handle_connect 0x7fd81406dee0 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fd81c053640 con 0x7fd83413f710 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 10 client.4288.objecter ms_dispatch 0x7fd834000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.602+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd81c04ef80 con 0x7fd83413f710 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.603+0000 7fd82b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.603+0000 7fd82b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:26.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.877+0000 7fd83961e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.877+0000 7fd83961e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.877+0000 7fd83961e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:26.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.877+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8340720c0 con 0x7fd83413f710 2024-03-25T19:35:26.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.905+0000 7fd82b7fe640 1 -- 172.21.15.29:0/776162455 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fd8340720c0 con 0x7fd83413f710 2024-03-25T19:35:26.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.905+0000 7fd82b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.905+0000 7fd82b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:26.905 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:26.905 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 10 librados: watch_flush enter 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 10 librados: watch_flush exit 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 >> v1:172.21.15.29:6800/2314944989 conn(0x7fd81406dee0 legacy=0x7fd8140703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 10 monclient: shutdown 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:26.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 >> v1:172.21.15.196:6789/0 conn(0x7fd83413f710 legacy=0x7fd83406e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 shutdown_connections 2024-03-25T19:35:26.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.907+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 >> 172.21.15.29:0/776162455 conn(0x7fd8340b7ea0 msgr2=0x7fd834074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:26.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.908+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 shutdown_connections 2024-03-25T19:35:26.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.908+0000 7fd83961e640 1 -- 172.21.15.29:0/776162455 wait complete. 2024-03-25T19:35:26.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:26.908+0000 7fd83961e640 1 librados: shutdown 2024-03-25T19:35:26.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:33.726 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:33.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 10 monclient: get_monmap_and_config 2024-03-25T19:35:33.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 10 monclient: monmap: 2024-03-25T19:35:33.843 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:33.843 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:33.844970+0000 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:33.844970+0000 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:33.844 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 1 Processor -- start 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 1 -- start start 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.844+0000 7fbdf109d640 10 monclient: init 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdec0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-c con 0x7fbdec13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-a con 0x7fbdec1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec142520 con 0x7fbdec1430c0 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0b28a0 con 0x7fbdec13f710 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0b2a30 con 0x7fbdec0a98f0 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.845+0000 7fbdf109d640 10 monclient(hunting): authenticate will time out at 6210.598728s 2024-03-25T19:35:33.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.851+0000 7fbdea59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbdec13f710 0x7fbdec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:47570/0 (socket says 172.21.15.29:47570) 2024-03-25T19:35:33.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.851+0000 7fbdea59c640 1 -- 172.21.15.29:0/3956001445 learned_addr learned my addr 172.21.15.29:0/3956001445 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:33.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508281603 0 0) 0x7fbdec0b28a0 con 0x7fbdec13f710 2024-03-25T19:35:33.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.865 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:35:33.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.866+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd4003520 con 0x7fbdec13f710 2024-03-25T19:35:33.874 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694099867 0 0) 0x7fbdec0b2a30 con 0x7fbdec0a98f0 2024-03-25T19:35:33.874 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:35:33.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.875+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec0b28a0 con 0x7fbdec0a98f0 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269674582 0 0) 0x7fbdd4003520 con 0x7fbdec13f710 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 >> v1:172.21.15.29:6790/0 conn(0x7fbdec0a98f0 legacy=0x7fbdec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 >> v1:172.21.15.29:6789/0 conn(0x7fbdec1430c0 legacy=0x7fbdec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:35:33.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0b44c0 con 0x7fbdec13f710 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbde0001690 con 0x7fbdec13f710 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbdf109d640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbdf109d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.885+0000 7fbde9d9b640 10 monclient: dump: 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.886 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:33.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.886+0000 7fbdf109d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:33.899 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde00017f0 con 0x7fbdec13f710 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 1 -- 172.21.15.29:0/3956001445 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbde0001c00 con 0x7fbdec13f710 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbde9d9b640 10 monclient: dump: 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:33.900 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbdf109d640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.900+0000 7fbdf109d640 4 set_mon_vals no callback set 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 10 monclient: shutdown 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:33.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 1 -- 172.21.15.29:0/3956001445 >> v1:172.21.15.196:6789/0 conn(0x7fbdec13f710 legacy=0x7fbdec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 1 -- 172.21.15.29:0/3956001445 shutdown_connections 2024-03-25T19:35:33.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 1 -- 172.21.15.29:0/3956001445 >> 172.21.15.29:0/3956001445 conn(0x7fbdec0b7ea0 msgr2=0x7fbdec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:33.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 1 -- 172.21.15.29:0/3956001445 shutdown_connections 2024-03-25T19:35:33.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.901+0000 7fbdf109d640 1 -- 172.21.15.29:0/3956001445 wait complete. 2024-03-25T19:35:33.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.902+0000 7fbdf109d640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.902+0000 7fbdf109d640 10 monclient: monmap: 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 librados: starting msgr at 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 librados: starting objecter 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 Processor -- start 2024-03-25T19:35:33.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 -- start start 2024-03-25T19:35:33.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 librados: setting wanted keys 2024-03-25T19:35:33.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 1 librados: calling monclient init 2024-03-25T19:35:33.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.903+0000 7fbdf109d640 10 monclient: init 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-a con 0x7fbdec0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-c con 0x7fbdec13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdec1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec1d77a0 con 0x7fbdec0a98f0 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec138e40 con 0x7fbdec1430c0 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0aba30 con 0x7fbdec13f710 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.904+0000 7fbdf109d640 10 monclient(hunting): authenticate will time out at 6210.657352s 2024-03-25T19:35:33.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.924+0000 7fbdead9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbdec0a98f0 0x7fbdec138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54448/0 (socket says 172.21.15.29:54448) 2024-03-25T19:35:33.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.924+0000 7fbdead9d640 1 -- 172.21.15.29:0/255898858 learned_addr learned my addr 172.21.15.29:0/255898858 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:33.927 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.927+0000 7fbdd37fe640 10 client.?.objecter ms_handle_connect 0x7fbdec1430c0 2024-03-25T19:35:33.927 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.927+0000 7fbdd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.927 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.927+0000 7fbdd37fe640 10 client.?.objecter ms_handle_connect 0x7fbdec0a98f0 2024-03-25T19:35:33.927 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.928+0000 7fbdd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.935+0000 7fbdd37fe640 10 client.?.objecter ms_handle_connect 0x7fbdec13f710 2024-03-25T19:35:33.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.935+0000 7fbdd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.939+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095307933 0 0) 0x7fbdec138e40 con 0x7fbdec1430c0 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.940+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.940+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.940+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.940+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:35:33.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.940+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdec1430c0 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.949+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2264066327 0 0) 0x7fbdd8004eb0 con 0x7fbdec1430c0 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdec138e40 con 0x7fbdec1430c0 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbde4001b20 con 0x7fbdec1430c0 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:33.949 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:33.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.950+0000 7fbdd37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048986921 0 0) 0x7fbdec1d77a0 con 0x7fbdec0a98f0 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:35:33.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.953+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdec0a98f0 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.959+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725509643 0 0) 0x7fbdec0aba30 con 0x7fbdec13f710 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.959+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.959+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.959+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.959+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:35:33.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.960+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec1d77a0 con 0x7fbdec13f710 2024-03-25T19:35:33.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3416061500 0 0) 0x7fbdec138e40 con 0x7fbdec1430c0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 >> v1:172.21.15.29:6790/0 conn(0x7fbdec13f710 legacy=0x7fbdec1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 >> v1:172.21.15.29:6789/0 conn(0x7fbdec0a98f0 legacy=0x7fbdec138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 1 monclient: found mon.b 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec1d8340 con 0x7fbdec1430c0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 10 monclient: _renew_subs 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec134820 con 0x7fbdec1430c0 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:33.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 10 monclient: _renew_subs 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec1380c0 con 0x7fbdec1430c0 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.969+0000 7fbdf109d640 1 librados: init done 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.971+0000 7fbdf109d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.971+0000 7fbdf109d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.971+0000 7fbdf109d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:33.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:33.971+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec0627b0 con 0x7fbdec1430c0 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde4002070 con 0x7fbdec1430c0 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbde4003b10 con 0x7fbdec1430c0 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbdd37fe640 10 monclient: dump: 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.251+0000 7fbde959a640 4 set_mon_vals no callback set 2024-03-25T19:35:34.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.252+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbde4003cf0 con 0x7fbdec1430c0 2024-03-25T19:35:34.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.253+0000 7fbdead9d640 10 monclient: get_auth_request con 0x7fbdd806dea0 auth_method 0 2024-03-25T19:35:34.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.253+0000 7fbdd37fe640 10 client.4294.objecter ms_handle_connect 0x7fbdd806dea0 2024-03-25T19:35:34.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fbde40538d0 con 0x7fbdec1430c0 2024-03-25T19:35:34.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 10 client.4294.objecter ms_dispatch 0x7fbdec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:34.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:34.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:34.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:35:34.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbde404f210 con 0x7fbdec1430c0 2024-03-25T19:35:34.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.257+0000 7fbdd37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:34.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.474+0000 7fbdf109d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.474+0000 7fbdf109d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.474+0000 7fbdf109d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:34.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.474+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdec14e8b0 con 0x7fbdec1430c0 2024-03-25T19:35:34.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.505+0000 7fbdd37fe640 1 -- 172.21.15.29:0/255898858 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fbde4051310 con 0x7fbdec1430c0 2024-03-25T19:35:34.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.505+0000 7fbdd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.505+0000 7fbdd37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:34.505 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:34.505 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:34.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.507+0000 7fbdf109d640 10 librados: watch_flush enter 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.507+0000 7fbdf109d640 10 librados: watch_flush exit 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.507+0000 7fbdf109d640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.507+0000 7fbdf109d640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbdd806dea0 legacy=0x7fbdd8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 10 monclient: shutdown 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 >> v1:172.21.15.196:6789/0 conn(0x7fbdec1430c0 legacy=0x7fbdec134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 shutdown_connections 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 >> 172.21.15.29:0/255898858 conn(0x7fbdec0b7ea0 msgr2=0x7fbdec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 shutdown_connections 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 -- 172.21.15.29:0/255898858 wait complete. 2024-03-25T19:35:34.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:34.508+0000 7fbdf109d640 1 librados: shutdown 2024-03-25T19:35:34.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:41.325 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:41.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.444+0000 7f231dcc1640 10 monclient: get_monmap_and_config 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.444+0000 7f231dcc1640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.444+0000 7f231dcc1640 10 monclient: monmap: 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:41.445381+0000 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:41.445381+0000 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:41.444 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:41.445 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.445+0000 7f231dcc1640 1 Processor -- start 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.445+0000 7f231dcc1640 1 -- start start 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.445+0000 7f231dcc1640 10 monclient: init 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f23180a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f231813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f23181430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318142520 con 0x7f231813f710 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180b28a0 con 0x7f23181430c0 2024-03-25T19:35:41.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180b2a30 con 0x7f23180a98f0 2024-03-25T19:35:41.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.446+0000 7f231dcc1640 10 monclient(hunting): authenticate will time out at 6218.199144s 2024-03-25T19:35:41.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.457+0000 7f230effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f231813f710 0x7f23181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53316/0 (socket says 172.21.15.29:53316) 2024-03-25T19:35:41.457 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.457+0000 7f230effd640 1 -- 172.21.15.29:0/1433395758 learned_addr learned my addr 172.21.15.29:0/1433395758 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.463+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864946904 0 0) 0x7f2318142520 con 0x7f231813f710 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.464+0000 7f2316ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.464+0000 7f2316ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.464+0000 7f2316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.464+0000 7f2316ffd640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:35:41.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.464+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003520 con 0x7f231813f710 2024-03-25T19:35:41.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246076266 0 0) 0x7f23180b28a0 con 0x7f23181430c0 2024-03-25T19:35:41.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:35:41.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.469+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318142520 con 0x7f23181430c0 2024-03-25T19:35:41.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312579072 0 0) 0x7f2304003520 con 0x7f231813f710 2024-03-25T19:35:41.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 >> v1:172.21.15.29:6790/0 conn(0x7f23180a98f0 legacy=0x7f23180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 >> v1:172.21.15.196:6789/0 conn(0x7f23181430c0 legacy=0x7f231814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 monclient: found mon.noname-a 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180b44c0 con 0x7f231813f710 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2300001690 con 0x7f231813f710 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f2316ffd640 10 monclient: dump: 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.471 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f231dcc1640 5 monclient: authenticate success, global_id 4299 2024-03-25T19:35:41.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.470+0000 7f231dcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23000017f0 con 0x7f231813f710 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 1 -- 172.21.15.29:0/1433395758 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2300001c00 con 0x7f231813f710 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f231dcc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f231dcc1640 4 set_mon_vals no callback set 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.492+0000 7f2316ffd640 10 monclient: dump: 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.492 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:41.493 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 10 monclient: shutdown 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 1 -- 172.21.15.29:0/1433395758 >> v1:172.21.15.29:6789/0 conn(0x7f231813f710 legacy=0x7f23181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 1 -- 172.21.15.29:0/1433395758 shutdown_connections 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 1 -- 172.21.15.29:0/1433395758 >> 172.21.15.29:0/1433395758 conn(0x7f23180b7ea0 msgr2=0x7f23180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 1 -- 172.21.15.29:0/1433395758 shutdown_connections 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.493+0000 7f231dcc1640 1 -- 172.21.15.29:0/1433395758 wait complete. 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.494+0000 7f231dcc1640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.494+0000 7f231dcc1640 10 monclient: monmap: 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.494+0000 7f231dcc1640 1 librados: starting msgr at 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.494+0000 7f231dcc1640 1 librados: starting objecter 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.494+0000 7f231dcc1640 1 Processor -- start 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 -- start start 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 librados: setting wanted keys 2024-03-25T19:35:41.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 librados: calling monclient init 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient: init 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f23180a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f231813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f23181430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318137110 con 0x7f231813f710 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23181319b0 con 0x7f23181430c0 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318135590 con 0x7f23180a98f0 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.495+0000 7f231dcc1640 10 monclient(hunting): authenticate will time out at 6218.248593s 2024-03-25T19:35:41.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.507+0000 7f230effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f231813f710 0x7f2318130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53360/0 (socket says 172.21.15.29:53360) 2024-03-25T19:35:41.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.507+0000 7f230effd640 1 -- 172.21.15.29:0/2100473874 learned_addr learned my addr 172.21.15.29:0/2100473874 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:41.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.509+0000 7f2314ff9640 10 client.?.objecter ms_handle_connect 0x7f231813f710 2024-03-25T19:35:41.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.509+0000 7f2314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.511+0000 7f2314ff9640 10 client.?.objecter ms_handle_connect 0x7f23181430c0 2024-03-25T19:35:41.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.511+0000 7f2314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.511+0000 7f2314ff9640 10 client.?.objecter ms_handle_connect 0x7f23180a98f0 2024-03-25T19:35:41.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.511+0000 7f2314ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518941947 0 0) 0x7f2318137110 con 0x7f231813f710 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:35:41.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.516+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f8004eb0 con 0x7f231813f710 2024-03-25T19:35:41.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683384500 0 0) 0x7f23181319b0 con 0x7f23181430c0 2024-03-25T19:35:41.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:35:41.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.523+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318137110 con 0x7f23181430c0 2024-03-25T19:35:41.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221174763 0 0) 0x7f2318135590 con 0x7f23180a98f0 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.527+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23181319b0 con 0x7f23180a98f0 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033615227 0 0) 0x7f22f8004eb0 con 0x7f231813f710 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2318135590 con 0x7f231813f710 2024-03-25T19:35:41.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2300001bc0 con 0x7f231813f710 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:41.528 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:41.529 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.528+0000 7f2314ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506558286 0 0) 0x7f2318137110 con 0x7f23181430c0 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f8004eb0 con 0x7f23181430c0 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2310001bf0 con 0x7f23181430c0 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.532+0000 7f2314ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:41.532 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:41.533 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:41.533 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 635732201 0 0) 0x7f2318135590 con 0x7f231813f710 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 >> v1:172.21.15.29:6790/0 conn(0x7f23180a98f0 legacy=0x7f23181d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 >> v1:172.21.15.196:6789/0 conn(0x7f23181430c0 legacy=0x7f23181312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 1 monclient: found mon.a 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.539+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231813b640 con 0x7f231813f710 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f2314ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f2314ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f2314ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 10 monclient: _renew_subs 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231813b890 con 0x7f231813f710 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 10 monclient: _renew_subs 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23181d8040 con 0x7f231813f710 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.540+0000 7f231dcc1640 1 librados: init done 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.541+0000 7f231dcc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.541+0000 7f231dcc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.541+0000 7f231dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.541+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23180627b0 con 0x7f231813f710 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23000011f0 con 0x7f231813f710 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2300004cb0 con 0x7f231813f710 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f2314ff9640 10 monclient: dump: 2024-03-25T19:35:41.884 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:41.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.884+0000 7f23167fc640 4 set_mon_vals no callback set 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.885+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f230001ad00 con 0x7f231813f710 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.886+0000 7f23177fe640 10 monclient: get_auth_request con 0x7f22f806de30 auth_method 0 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.886+0000 7f2314ff9640 10 client.4302.objecter ms_handle_connect 0x7f22f806de30 2024-03-25T19:35:41.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f23000549f0 con 0x7f231813f710 2024-03-25T19:35:41.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 10 client.4302.objecter ms_dispatch 0x7f2318000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2300050330 con 0x7f231813f710 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:41.891+0000 7f2314ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:42.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.106+0000 7f231dcc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.106+0000 7f231dcc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.106+0000 7f231dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:42.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.106+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23181352b0 con 0x7f231813f710 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.137+0000 7f2314ff9640 1 -- 172.21.15.29:0/2100473874 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f23181352b0 con 0x7f231813f710 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.138+0000 7f2314ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.138+0000 7f2314ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 10 librados: watch_flush enter 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 10 librados: watch_flush exit 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 10 client.4302.objecter successfully canceled tick 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 >> v1:172.21.15.29:6800/2314944989 conn(0x7f22f806de30 legacy=0x7f22f80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 10 monclient: shutdown 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 >> v1:172.21.15.29:6789/0 conn(0x7f231813f710 legacy=0x7f2318130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 shutdown_connections 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 >> 172.21.15.29:0/2100473874 conn(0x7f23180b7ea0 msgr2=0x7f2318074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 shutdown_connections 2024-03-25T19:35:42.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 -- 172.21.15.29:0/2100473874 wait complete. 2024-03-25T19:35:42.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:42.140+0000 7f231dcc1640 1 librados: shutdown 2024-03-25T19:35:42.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:44.493 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:14.656 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.500 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:14.663 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.546 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:14.709 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.550 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:14.714 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.562 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:14.724 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.567 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:14.730 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.578 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:14.742 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.585 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:14.748 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:44.609 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:14.771 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:48.957 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:49.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.072+0000 7f6e00f54640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.072+0000 7f6e00f54640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.072+0000 7f6e00f54640 10 monclient: monmap: 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:49.073009+0000 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:49.073009+0000 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:49.072 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.073+0000 7f6e00f54640 1 Processor -- start 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.073+0000 7f6e00f54640 1 -- start start 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.073+0000 7f6e00f54640 10 monclient: init 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-c con 0x7f6dfc0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-a con 0x7f6dfc13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:49.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-b con 0x7f6dfc1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:49.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc142520 con 0x7f6dfc13f710 2024-03-25T19:35:49.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0b28a0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0b2a30 con 0x7f6dfc1430c0 2024-03-25T19:35:49.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.074+0000 7f6e00f54640 10 monclient(hunting): authenticate will time out at 6225.827134s 2024-03-25T19:35:49.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.076+0000 7f6dfa59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6dfc0a98f0 0x7f6dfc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:33226/0 (socket says 172.21.15.29:33226) 2024-03-25T19:35:49.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.076+0000 7f6dfa59c640 1 -- 172.21.15.29:0/878906484 learned_addr learned my addr 172.21.15.29:0/878906484 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:49.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782052773 0 0) 0x7f6dfc0b28a0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:35:49.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.096+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4003520 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948993081 0 0) 0x7f6dfc142520 con 0x7f6dfc13f710 2024-03-25T19:35:49.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:35:49.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc0b28a0 con 0x7f6dfc13f710 2024-03-25T19:35:49.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822194960 0 0) 0x7f6de4003520 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 >> v1:172.21.15.29:6790/0 conn(0x7f6dfc1430c0 legacy=0x7f6dfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 >> v1:172.21.15.29:6789/0 conn(0x7f6dfc13f710 legacy=0x7f6dfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 monclient: found mon.noname-c 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc0b44c0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6dec001690 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6df959a640 10 monclient: dump: 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6e00f54640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:35:49.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.101+0000 7f6e00f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec001870 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 1 -- 172.21.15.29:0/878906484 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6dec001c80 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6e00f54640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6e00f54640 4 set_mon_vals no callback set 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.111+0000 7f6df959a640 10 monclient: dump: 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.111 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.112 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 10 monclient: shutdown 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 1 -- 172.21.15.29:0/878906484 >> v1:172.21.15.196:6789/0 conn(0x7f6dfc0a98f0 legacy=0x7f6dfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 1 -- 172.21.15.29:0/878906484 shutdown_connections 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 1 -- 172.21.15.29:0/878906484 >> 172.21.15.29:0/878906484 conn(0x7f6dfc0b7ea0 msgr2=0x7f6dfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 1 -- 172.21.15.29:0/878906484 shutdown_connections 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.112+0000 7f6e00f54640 1 -- 172.21.15.29:0/878906484 wait complete. 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.113+0000 7f6e00f54640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.113+0000 7f6e00f54640 10 monclient: monmap: 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.113+0000 7f6e00f54640 1 librados: starting msgr at 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.113+0000 7f6e00f54640 1 librados: starting objecter 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 1 Processor -- start 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 1 -- start start 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 1 librados: setting wanted keys 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 1 librados: calling monclient init 2024-03-25T19:35:49.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 10 monclient: init 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.114+0000 7f6e00f54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-b con 0x7f6dfc0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-c con 0x7f6dfc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 10 monclient(hunting): picked mon.noname-a con 0x7f6dfc1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc139ee0 con 0x7f6dfc1430c0 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0b61a0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc13b540 con 0x7f6dfc13f710 2024-03-25T19:35:49.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.115+0000 7f6e00f54640 10 monclient(hunting): authenticate will time out at 6225.868478s 2024-03-25T19:35:49.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.117+0000 7f6dfa59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6dfc0a98f0 0x7f6dfc0b5a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:33260/0 (socket says 172.21.15.29:33260) 2024-03-25T19:35:49.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.117+0000 7f6dfa59c640 1 -- 172.21.15.29:0/3244661335 learned_addr learned my addr 172.21.15.29:0/3244661335 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:49.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.119+0000 7f6deaffd640 10 client.?.objecter ms_handle_connect 0x7f6dfc0a98f0 2024-03-25T19:35:49.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.119+0000 7f6deaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.125 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.125+0000 7f6deaffd640 10 client.?.objecter ms_handle_connect 0x7f6dfc13f710 2024-03-25T19:35:49.125 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.125+0000 7f6deaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907164691 0 0) 0x7f6dfc0b61a0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:35:49.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.129+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc004eb0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.131+0000 7f6deaffd640 10 client.?.objecter ms_handle_connect 0x7f6dfc1430c0 2024-03-25T19:35:49.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.131+0000 7f6deaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882258182 0 0) 0x7f6dfc13b540 con 0x7f6dfc13f710 2024-03-25T19:35:49.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:35:49.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.137+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc0b61a0 con 0x7f6dfc13f710 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1967917711 0 0) 0x7f6ddc004eb0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dfc13b540 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6dec001bc0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.141 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.142 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.142 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.141+0000 7f6deaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227328659 0 0) 0x7f6dfc139ee0 con 0x7f6dfc1430c0 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:35:49.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.145+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc004eb0 con 0x7f6dfc1430c0 2024-03-25T19:35:49.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120650609 0 0) 0x7f6ddc004eb0 con 0x7f6dfc1430c0 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dfc139ee0 con 0x7f6dfc1430c0 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6df4001a10 con 0x7f6dfc1430c0 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.155+0000 7f6deaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.155 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.156 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.156 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998575934 0 0) 0x7f6dfc0b61a0 con 0x7f6dfc13f710 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ddc004eb0 con 0x7f6dfc13f710 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6df00018b0 con 0x7f6dfc13f710 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.161 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1037316876 0 0) 0x7f6dfc13b540 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 >> v1:172.21.15.29:6790/0 conn(0x7f6dfc13f710 legacy=0x7f6dfc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 >> v1:172.21.15.29:6789/0 conn(0x7f6dfc1430c0 legacy=0x7f6dfc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 monclient: found mon.b 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc13b820 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6e00f54640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6e00f54640 10 monclient: _renew_subs 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6e00f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.160+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc1399a0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.161+0000 7f6e00f54640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.161+0000 7f6e00f54640 10 monclient: _renew_subs 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.161+0000 7f6e00f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.161+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc12f930 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.161+0000 7f6e00f54640 1 librados: init done 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.162+0000 7f6e00f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.162+0000 7f6e00f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.162+0000 7f6e00f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.162+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc0627b0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec004bf0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6df8d99640 4 set_mon_vals no callback set 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f6dec004f00 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.187+0000 7f6deaffd640 10 monclient: dump: 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:49.187 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:49.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.188+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f6dec01aed0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.189+0000 7f6df9d9b640 10 monclient: get_auth_request con 0x7f6ddc06e110 auth_method 0 2024-03-25T19:35:49.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.189+0000 7f6deaffd640 10 client.4306.objecter ms_handle_connect 0x7f6ddc06e110 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.459+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f6dec0549f0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.459+0000 7f6deaffd640 10 client.4306.objecter ms_dispatch 0x7f6dfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.459+0000 7f6deaffd640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.459+0000 7f6deaffd640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.459+0000 7f6deaffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.460+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6dec0503b0 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.460+0000 7f6deaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.460+0000 7f6deaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:49.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.709+0000 7f6e00f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.709+0000 7f6e00f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.709+0000 7f6e00f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:35:49.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.709+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dfc13e150 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.730+0000 7f6deaffd640 1 -- 172.21.15.29:0/3244661335 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f6dec052430 con 0x7f6dfc0a98f0 2024-03-25T19:35:49.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.731+0000 7f6deaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.731+0000 7f6deaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.730 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:49.731 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.732+0000 7f6e00f54640 10 librados: watch_flush enter 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 10 librados: watch_flush exit 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 10 client.4306.objecter successfully canceled tick 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 >> v1:172.21.15.29:6800/2314944989 conn(0x7f6ddc06e110 legacy=0x7f6ddc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 10 monclient: shutdown 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 >> v1:172.21.15.196:6789/0 conn(0x7f6dfc0a98f0 legacy=0x7f6dfc0b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 shutdown_connections 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 >> 172.21.15.29:0/3244661335 conn(0x7f6dfc0b7ea0 msgr2=0x7f6dfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 shutdown_connections 2024-03-25T19:35:49.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 -- 172.21.15.29:0/3244661335 wait complete. 2024-03-25T19:35:49.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:49.733+0000 7f6e00f54640 1 librados: shutdown 2024-03-25T19:35:49.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:56.552 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:56.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.669+0000 7fad2133d640 10 monclient: get_monmap_and_config 2024-03-25T19:35:56.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.669+0000 7fad2133d640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.669+0000 7fad2133d640 10 monclient: monmap: 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:35:56.670364+0000 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:35:56.670364+0000 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 1 Processor -- start 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 1 -- start start 2024-03-25T19:35:56.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 10 monclient: init 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-c con 0x7fad1c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.670+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c142520 con 0x7fad1c13f710 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b28a0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b2a30 con 0x7fad1c1430c0 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.671+0000 7fad2133d640 10 monclient(hunting): authenticate will time out at 6233.424064s 2024-03-25T19:35:56.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.673+0000 7fad1affd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fad1c0a98f0 0x7fad1c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:59064/0 (socket says 172.21.15.29:59064) 2024-03-25T19:35:56.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.673+0000 7fad1affd640 1 -- 172.21.15.29:0/1321373278 learned_addr learned my addr 172.21.15.29:0/1321373278 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206413795 0 0) 0x7fad1c0b28a0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:35:56.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.680+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad04003520 con 0x7fad1c0a98f0 2024-03-25T19:35:56.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987347965 0 0) 0x7fad04003520 con 0x7fad1c0a98f0 2024-03-25T19:35:56.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 >> v1:172.21.15.29:6790/0 conn(0x7fad1c1430c0 legacy=0x7fad1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 >> v1:172.21.15.29:6789/0 conn(0x7fad1c13f710 legacy=0x7fad1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0b44c0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad0c001690 con 0x7fad1c0a98f0 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad19ffb640 10 monclient: dump: 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:56.695 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad2133d640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:35:56.696 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.694+0000 7fad2133d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad0c0017f0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 1 -- 172.21.15.29:0/1321373278 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad0c001c00 con 0x7fad1c0a98f0 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad19ffb640 10 monclient: dump: 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.702 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad2133d640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:56.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.702+0000 7fad2133d640 4 set_mon_vals no callback set 2024-03-25T19:35:56.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 10 monclient: shutdown 2024-03-25T19:35:56.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:56.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 1 -- 172.21.15.29:0/1321373278 >> v1:172.21.15.196:6789/0 conn(0x7fad1c0a98f0 legacy=0x7fad1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 1 -- 172.21.15.29:0/1321373278 shutdown_connections 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 1 -- 172.21.15.29:0/1321373278 >> 172.21.15.29:0/1321373278 conn(0x7fad1c0b7ea0 msgr2=0x7fad1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 1 -- 172.21.15.29:0/1321373278 shutdown_connections 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.703+0000 7fad2133d640 1 -- 172.21.15.29:0/1321373278 wait complete. 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.704+0000 7fad2133d640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.704+0000 7fad2133d640 10 monclient: monmap: 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:35:56.706 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.704+0000 7fad2133d640 1 librados: starting msgr at 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.704+0000 7fad2133d640 1 librados: starting objecter 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 Processor -- start 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 -- start start 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 librados: setting wanted keys 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 librados: calling monclient init 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient: init 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:56.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-c con 0x7fad1c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c1db7d0 con 0x7fad1c1430c0 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.705+0000 7fad2133d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b1420 con 0x7fad1c0a98f0 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.706+0000 7fad2133d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b15b0 con 0x7fad1c13f710 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.706+0000 7fad2133d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.706+0000 7fad2133d640 10 monclient(hunting): authenticate will time out at 6233.458894s 2024-03-25T19:35:56.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.716+0000 7fad1b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fad1c1430c0 0x7fad1c1d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43052/0 (socket says 172.21.15.29:43052) 2024-03-25T19:35:56.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.716+0000 7fad1b7fe640 1 -- 172.21.15.29:0/1996011487 learned_addr learned my addr 172.21.15.29:0/1996011487 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:35:56.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.718+0000 7fad037fe640 10 client.?.objecter ms_handle_connect 0x7fad1c0a98f0 2024-03-25T19:35:56.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.718+0000 7fad037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.719+0000 7fad037fe640 10 client.?.objecter ms_handle_connect 0x7fad1c1430c0 2024-03-25T19:35:56.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.719+0000 7fad037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.729 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 client.?.objecter ms_handle_connect 0x7fad1c13f710 2024-03-25T19:35:56.729 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813732481 0 0) 0x7fad1c1db7d0 con 0x7fad1c1430c0 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:35:56.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.730+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad08004eb0 con 0x7fad1c1430c0 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320500559 0 0) 0x7fad1c0b1420 con 0x7fad1c0a98f0 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.735+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1c1db7d0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999279566 0 0) 0x7fad1c0b15b0 con 0x7fad1c13f710 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:35:56.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.736+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1c0b1420 con 0x7fad1c13f710 2024-03-25T19:35:56.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286032804 0 0) 0x7fad08004eb0 con 0x7fad1c1430c0 2024-03-25T19:35:56.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad1c0b15b0 con 0x7fad1c1430c0 2024-03-25T19:35:56.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad14001c90 con 0x7fad1c1430c0 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 10 monclient(hunting): dump: 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.748+0000 7fad037fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2020794961 0 0) 0x7fad1c1db7d0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad08004eb0 con 0x7fad1c0a98f0 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad0c001b00 con 0x7fad1c0a98f0 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:56.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.750+0000 7fad037fe640 10 monclient(hunting): dump: 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:56.751 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.757+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4212980208 0 0) 0x7fad1c0b1420 con 0x7fad1c13f710 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.757+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad1c1db7d0 con 0x7fad1c13f710 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.758+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad10001830 con 0x7fad1c13f710 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.758+0000 7fad037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.758+0000 7fad037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.758+0000 7fad037fe640 10 monclient(hunting): dump: 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:56.757 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:56.758 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:56.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488645185 0 0) 0x7fad1c0b15b0 con 0x7fad1c1430c0 2024-03-25T19:35:56.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 >> v1:172.21.15.29:6790/0 conn(0x7fad1c13f710 legacy=0x7fad1c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 >> v1:172.21.15.196:6789/0 conn(0x7fad1c0a98f0 legacy=0x7fad1c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 1 monclient: found mon.a 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0b1740 con 0x7fad1c1430c0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad2133d640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.765+0000 7fad2133d640 10 monclient: _renew_subs 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c1db1a0 con 0x7fad1c1430c0 2024-03-25T19:35:56.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 10 monclient: _renew_subs 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c0b08f0 con 0x7fad1c1430c0 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 1 librados: init done 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.766+0000 7fad2133d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.767+0000 7fad2133d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.767+0000 7fad2133d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:56.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:56.767+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c0627b0 con 0x7fad1c1430c0 2024-03-25T19:35:57.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad14003990 con 0x7fad1c1430c0 2024-03-25T19:35:57.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fad14003ce0 con 0x7fad1c1430c0 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad037fe640 10 monclient: dump: 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:35:57.082 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:35:57.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.082+0000 7fad197fa640 4 set_mon_vals no callback set 2024-03-25T19:35:57.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.083+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fad14003f00 con 0x7fad1c1430c0 2024-03-25T19:35:57.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.083+0000 7fad1affd640 10 monclient: get_auth_request con 0x7fad0806e0f0 auth_method 0 2024-03-25T19:35:57.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.083+0000 7fad037fe640 10 client.4314.objecter ms_handle_connect 0x7fad0806e0f0 2024-03-25T19:35:57.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fad14054b10 con 0x7fad1c1430c0 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 10 client.4314.objecter ms_dispatch 0x7fad1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad14050620 con 0x7fad1c1430c0 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.088+0000 7fad037fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:57.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.347+0000 7fad2133d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.347+0000 7fad2133d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.347+0000 7fad2133d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.347+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad1c0abae0 con 0x7fad1c1430c0 2024-03-25T19:35:57.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.383+0000 7fad037fe640 1 -- 172.21.15.29:0/1996011487 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fad14052550 con 0x7fad1c1430c0 2024-03-25T19:35:57.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.383+0000 7fad037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.383+0000 7fad037fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:57.383 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:35:57.383 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:57.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.385+0000 7fad2133d640 10 librados: watch_flush enter 2024-03-25T19:35:57.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 10 librados: watch_flush exit 2024-03-25T19:35:57.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 10 client.4314.objecter successfully canceled tick 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 >> v1:172.21.15.29:6800/2314944989 conn(0x7fad0806e0f0 legacy=0x7fad080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 10 monclient: shutdown 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 >> v1:172.21.15.29:6789/0 conn(0x7fad1c1430c0 legacy=0x7fad1c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 shutdown_connections 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 >> 172.21.15.29:0/1996011487 conn(0x7fad1c0b7ea0 msgr2=0x7fad1c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 shutdown_connections 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 -- 172.21.15.29:0/1996011487 wait complete. 2024-03-25T19:35:57.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:35:57.386+0000 7fad2133d640 1 librados: shutdown 2024-03-25T19:35:57.398 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:04.204 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.319+0000 7fec3cd58640 10 monclient: get_monmap_and_config 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.319+0000 7fec3cd58640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.319+0000 7fec3cd58640 10 monclient: monmap: 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:04.320566+0000 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:04.320566+0000 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.319 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:04.320 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:04.320 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:04.320 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.320+0000 7fec3cd58640 1 Processor -- start 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.320+0000 7fec3cd58640 1 -- start start 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.320+0000 7fec3cd58640 10 monclient: init 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-c con 0x7fec380b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-b con 0x7fec380b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-a con 0x7fec3814be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380af940 con 0x7fec3814be90 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380b0a70 con 0x7fec380b4d60 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380b0d50 con 0x7fec380b1730 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.321+0000 7fec3cd58640 10 monclient(hunting): authenticate will time out at 6241.074216s 2024-03-25T19:36:04.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.330+0000 7fec35d9b640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fec380b1730 0x7fec380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:39982/0 (socket says 172.21.15.29:39982) 2024-03-25T19:36:04.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.330+0000 7fec35d9b640 1 -- 172.21.15.29:0/2609521697 learned_addr learned my addr 172.21.15.29:0/2609521697 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:04.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095652295 0 0) 0x7fec380b0d50 con 0x7fec380b1730 2024-03-25T19:36:04.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:36:04.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.340+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec20003520 con 0x7fec380b1730 2024-03-25T19:36:04.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.348+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885422723 0 0) 0x7fec20003520 con 0x7fec380b1730 2024-03-25T19:36:04.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 >> v1:172.21.15.196:6789/0 conn(0x7fec380b4d60 legacy=0x7fec3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 >> v1:172.21.15.29:6789/0 conn(0x7fec3814be90 legacy=0x7fec3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 1 monclient: found mon.noname-b 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380b1030 con 0x7fec380b1730 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec2c001690 con 0x7fec380b1730 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3559a640 10 monclient: dump: 2024-03-25T19:36:04.349 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3cd58640 5 monclient: authenticate success, global_id 4316 2024-03-25T19:36:04.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.349+0000 7fec3cd58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:04.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec2c0017f0 con 0x7fec380b1730 2024-03-25T19:36:04.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 1 -- 172.21.15.29:0/2609521697 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec2c001c40 con 0x7fec380b1730 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3cd58640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3cd58640 4 set_mon_vals no callback set 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3559a640 10 monclient: dump: 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.379 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.380 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.380 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.380 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.380 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3cd58640 10 monclient: shutdown 2024-03-25T19:36:04.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3cd58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.379+0000 7fec3cd58640 1 -- 172.21.15.29:0/2609521697 >> v1:172.21.15.29:6790/0 conn(0x7fec380b1730 legacy=0x7fec380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.380+0000 7fec3cd58640 1 -- 172.21.15.29:0/2609521697 shutdown_connections 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.380+0000 7fec3cd58640 1 -- 172.21.15.29:0/2609521697 >> 172.21.15.29:0/2609521697 conn(0x7fec380617a0 msgr2=0x7fec38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.380+0000 7fec3cd58640 1 -- 172.21.15.29:0/2609521697 shutdown_connections 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.380+0000 7fec3cd58640 1 -- 172.21.15.29:0/2609521697 wait complete. 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.381+0000 7fec3cd58640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.381+0000 7fec3cd58640 10 monclient: monmap: 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:04.383 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.381+0000 7fec3cd58640 1 librados: starting msgr at 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.381+0000 7fec3cd58640 1 librados: starting objecter 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.381+0000 7fec3cd58640 1 Processor -- start 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 -- start start 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 librados: setting wanted keys 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 librados: calling monclient init 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient: init 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:04.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-b con 0x7fec380b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-c con 0x7fec380b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient(hunting): picked mon.noname-a con 0x7fec3814be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec38145600 con 0x7fec3814be90 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec38140520 con 0x7fec380b1730 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3813d800 con 0x7fec380b4d60 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.385 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.382+0000 7fec3cd58640 10 monclient(hunting): authenticate will time out at 6241.135744s 2024-03-25T19:36:04.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.389+0000 7fec35d9b640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fec380b4d60 0x7fec3812fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:40022/0 (socket says 172.21.15.29:40022) 2024-03-25T19:36:04.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.389+0000 7fec35d9b640 1 -- 172.21.15.29:0/124855489 learned_addr learned my addr 172.21.15.29:0/124855489 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:04.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.390+0000 7fec26ffd640 10 client.?.objecter ms_handle_connect 0x7fec380b4d60 2024-03-25T19:36:04.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.390+0000 7fec26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.397+0000 7fec26ffd640 10 client.?.objecter ms_handle_connect 0x7fec380b1730 2024-03-25T19:36:04.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.397+0000 7fec26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.399+0000 7fec26ffd640 10 client.?.objecter ms_handle_connect 0x7fec3814be90 2024-03-25T19:36:04.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.399+0000 7fec26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244103850 0 0) 0x7fec3813d800 con 0x7fec380b4d60 2024-03-25T19:36:04.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.402 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:36:04.402 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.402+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec18004eb0 con 0x7fec380b4d60 2024-03-25T19:36:04.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528309897 0 0) 0x7fec38140520 con 0x7fec380b1730 2024-03-25T19:36:04.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:36:04.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.411+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3813d800 con 0x7fec380b1730 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697642062 0 0) 0x7fec38145600 con 0x7fec3814be90 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:36:04.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.413+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec38140520 con 0x7fec3814be90 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994214533 0 0) 0x7fec3813d800 con 0x7fec380b1730 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec38145600 con 0x7fec380b1730 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec28001c30 con 0x7fec380b1730 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.419 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.419+0000 7fec26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292137550 0 0) 0x7fec18004eb0 con 0x7fec380b4d60 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec3813d800 con 0x7fec380b4d60 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec2c001b00 con 0x7fec380b4d60 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:04.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.427+0000 7fec26ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.427 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559994629 0 0) 0x7fec38140520 con 0x7fec3814be90 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec18004eb0 con 0x7fec3814be90 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec30001710 con 0x7fec3814be90 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.428+0000 7fec26ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.428 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.429 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1195352593 0 0) 0x7fec38145600 con 0x7fec380b1730 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 >> v1:172.21.15.29:6790/0 conn(0x7fec380b4d60 legacy=0x7fec3812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 >> v1:172.21.15.29:6789/0 conn(0x7fec3814be90 legacy=0x7fec381e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 1 monclient: found mon.b 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec381448a0 con 0x7fec380b1730 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 10 monclient: _renew_subs 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3813eeb0 con 0x7fec380b1730 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 10 monclient: _renew_subs 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec381444d0 con 0x7fec380b1730 2024-03-25T19:36:04.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.435+0000 7fec3cd58640 1 librados: init done 2024-03-25T19:36:04.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.436+0000 7fec3cd58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.436+0000 7fec3cd58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.436+0000 7fec3cd58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:04.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.436+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec381401f0 con 0x7fec380b1730 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec28003970 con 0x7fec380b1730 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fec28003dc0 con 0x7fec380b1730 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec26ffd640 10 monclient: dump: 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:04.710 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.710+0000 7fec34d99640 4 set_mon_vals no callback set 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.711+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fec28003fe0 con 0x7fec380b1730 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.712+0000 7fec35d9b640 10 monclient: get_auth_request con 0x7fec1806e0f0 auth_method 0 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.712+0000 7fec26ffd640 10 client.4318.objecter ms_handle_connect 0x7fec1806e0f0 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.716+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fec280538c0 con 0x7fec380b1730 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.716+0000 7fec26ffd640 10 client.4318.objecter ms_dispatch 0x7fec38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.716+0000 7fec26ffd640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.716+0000 7fec26ffd640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.716+0000 7fec26ffd640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.717+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec2804f3d0 con 0x7fec380b1730 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.717+0000 7fec26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.717+0000 7fec26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:04.853 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:35.017 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.863 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:35.027 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.895 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:35.059 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.909 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:35.071 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.914 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:35.076 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.920 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:35.084 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.941 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:35.104 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.945 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:35.109 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.956+0000 7fec3cd58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.956+0000 7fec3cd58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.956+0000 7fec3cd58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:04.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:04.956+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec3813f040 con 0x7fec380b1730 2024-03-25T19:36:04.971 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:35.135 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:05.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.004+0000 7fec26ffd640 1 -- 172.21.15.29:0/124855489 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fec28051300 con 0x7fec380b1730 2024-03-25T19:36:05.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.004+0000 7fec26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.004+0000 7fec26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:05.004 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:05.004 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 10 librados: watch_flush enter 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 10 librados: watch_flush exit 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 10 client.4318.objecter successfully canceled tick 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 >> v1:172.21.15.29:6800/2314944989 conn(0x7fec1806e0f0 legacy=0x7fec180705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 10 monclient: shutdown 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 >> v1:172.21.15.196:6789/0 conn(0x7fec380b1730 legacy=0x7fec3813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 shutdown_connections 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 >> 172.21.15.29:0/124855489 conn(0x7fec380617a0 msgr2=0x7fec380b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 shutdown_connections 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 -- 172.21.15.29:0/124855489 wait complete. 2024-03-25T19:36:05.008 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:05.007+0000 7fec3cd58640 1 librados: shutdown 2024-03-25T19:36:05.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:11.826 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.941+0000 7fd492a7b640 10 monclient: get_monmap_and_config 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.941+0000 7fd492a7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.941+0000 7fd492a7b640 10 monclient: monmap: 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:11.941996+0000 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:11.941996+0000 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:11.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.941+0000 7fd492a7b640 1 Processor -- start 2024-03-25T19:36:11.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.942+0000 7fd492a7b640 1 -- start start 2024-03-25T19:36:11.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.942+0000 7fd492a7b640 10 monclient: init 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.942+0000 7fd492a7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.942+0000 7fd492a7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd48c0b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd48c0b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd48c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0af940 con 0x7fd48c0b1730 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0b0a70 con 0x7fd48c0b4d60 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0b0d50 con 0x7fd48c14be90 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.943+0000 7fd492a7b640 10 monclient(hunting): authenticate will time out at 6248.696125s 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.945+0000 7fd490817640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd48c0b4d60 0x7fd48c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:43590/0 (socket says 172.21.15.29:43590) 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.945+0000 7fd490817640 1 -- 172.21.15.29:0/1503314162 learned_addr learned my addr 172.21.15.29:0/1503314162 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761975322 0 0) 0x7fd48c0b0a70 con 0x7fd48c0b4d60 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:36:11.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.956+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c003520 con 0x7fd48c0b4d60 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871548142 0 0) 0x7fd47c003520 con 0x7fd48c0b4d60 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 >> v1:172.21.15.29:6790/0 conn(0x7fd48c14be90 legacy=0x7fd48c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 >> v1:172.21.15.29:6789/0 conn(0x7fd48c0b1730 legacy=0x7fd48c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0b1030 con 0x7fd48c0b4d60 2024-03-25T19:36:11.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd48b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd492a7b640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.965+0000 7fd492a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.966+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd480001690 con 0x7fd48c0b4d60 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.966+0000 7fd48b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.966+0000 7fd48b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.966+0000 7fd48b7fe640 10 monclient: dump: 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:11.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.966+0000 7fd492a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd480001920 con 0x7fd48c0b4d60 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 1 -- 172.21.15.29:0/1503314162 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd480001c30 con 0x7fd48c0b4d60 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd48b7fe640 10 monclient: dump: 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:11.975 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd492a7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:11.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.975+0000 7fd492a7b640 4 set_mon_vals no callback set 2024-03-25T19:36:11.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.976+0000 7fd492a7b640 10 monclient: shutdown 2024-03-25T19:36:11.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.976+0000 7fd492a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.976+0000 7fd492a7b640 1 -- 172.21.15.29:0/1503314162 >> v1:172.21.15.196:6789/0 conn(0x7fd48c0b4d60 legacy=0x7fd48c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.976+0000 7fd492a7b640 1 -- 172.21.15.29:0/1503314162 shutdown_connections 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.976+0000 7fd492a7b640 1 -- 172.21.15.29:0/1503314162 >> 172.21.15.29:0/1503314162 conn(0x7fd48c0617a0 msgr2=0x7fd48c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.977+0000 7fd492a7b640 1 -- 172.21.15.29:0/1503314162 shutdown_connections 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.977+0000 7fd492a7b640 1 -- 172.21.15.29:0/1503314162 wait complete. 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.977+0000 7fd492a7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.977+0000 7fd492a7b640 10 monclient: monmap: 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:11.978 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 librados: starting msgr at 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 librados: starting objecter 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 Processor -- start 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 -- start start 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 librados: setting wanted keys 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 1 librados: calling monclient init 2024-03-25T19:36:11.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 10 monclient: init 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.978+0000 7fd492a7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd48c0b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd48c0b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd48c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c1d79d0 con 0x7fd48c0b1730 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c141f00 con 0x7fd48c14be90 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c140f70 con 0x7fd48c0b4d60 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.979+0000 7fd492a7b640 10 monclient(hunting): authenticate will time out at 6248.732182s 2024-03-25T19:36:11.983 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.984+0000 7fd48bfff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fd48c0b4d60 0x7fd48c1d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:33310/0 (socket says 172.21.15.29:33310) 2024-03-25T19:36:11.983 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.984+0000 7fd48bfff640 1 -- 172.21.15.29:0/2587147526 learned_addr learned my addr 172.21.15.29:0/2587147526 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:11.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.987+0000 7fd4897fa640 10 client.?.objecter ms_handle_connect 0x7fd48c0b4d60 2024-03-25T19:36:11.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.987+0000 7fd4897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.990 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.991+0000 7fd4897fa640 10 client.?.objecter ms_handle_connect 0x7fd48c0b1730 2024-03-25T19:36:11.990 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.991+0000 7fd4897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456496363 0 0) 0x7fd48c140f70 con 0x7fd48c0b4d60 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:36:11.994 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.994+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd470004eb0 con 0x7fd48c0b4d60 2024-03-25T19:36:11.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624585662 0 0) 0x7fd48c1d79d0 con 0x7fd48c0b1730 2024-03-25T19:36:11.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:36:11.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.996+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd48c140f70 con 0x7fd48c0b1730 2024-03-25T19:36:11.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.997+0000 7fd4897fa640 10 client.?.objecter ms_handle_connect 0x7fd48c14be90 2024-03-25T19:36:11.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:11.997+0000 7fd4897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2344696507 0 0) 0x7fd48c140f70 con 0x7fd48c0b1730 2024-03-25T19:36:12.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd48c1d79d0 con 0x7fd48c0b1730 2024-03-25T19:36:12.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd480001650 con 0x7fd48c0b1730 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 10 monclient(hunting): dump: 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:12.006 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:12.007 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.006+0000 7fd4897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079778519 0 0) 0x7fd48c141f00 con 0x7fd48c14be90 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.013+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd48c140f70 con 0x7fd48c14be90 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987437490 0 0) 0x7fd470004eb0 con 0x7fd48c0b4d60 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd48c141f00 con 0x7fd48c0b4d60 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd478001a30 con 0x7fd48c0b4d60 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.014+0000 7fd4897fa640 10 monclient(hunting): dump: 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:12.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 421189619 0 0) 0x7fd48c1d79d0 con 0x7fd48c0b1730 2024-03-25T19:36:12.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 >> v1:172.21.15.29:6790/0 conn(0x7fd48c0b4d60 legacy=0x7fd48c1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 >> v1:172.21.15.196:6789/0 conn(0x7fd48c14be90 legacy=0x7fd48c142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 1 monclient: found mon.a 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c141100 con 0x7fd48c0b1730 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd4897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 10 monclient: _renew_subs 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c1d8570 con 0x7fd48c0b1730 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 10 monclient: _renew_subs 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:12.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c1427e0 con 0x7fd48c0b1730 2024-03-25T19:36:12.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.021+0000 7fd492a7b640 1 librados: init done 2024-03-25T19:36:12.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.022+0000 7fd492a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.022+0000 7fd492a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.022+0000 7fd492a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:12.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.022+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c0627b0 con 0x7fd48c0b1730 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd480004a90 con 0x7fd48c0b1730 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd480004da0 con 0x7fd48c0b1730 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd4897fa640 10 monclient: dump: 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:12.346 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.346+0000 7fd48affd640 4 set_mon_vals no callback set 2024-03-25T19:36:12.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.347+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fd48001ad70 con 0x7fd48c0b1730 2024-03-25T19:36:12.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.348+0000 7fd48bfff640 10 monclient: get_auth_request con 0x7fd47006df70 auth_method 0 2024-03-25T19:36:12.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.348+0000 7fd4897fa640 10 client.4323.objecter ms_handle_connect 0x7fd47006df70 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fd4800548d0 con 0x7fd48c0b1730 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 10 client.4323.objecter ms_dispatch 0x7fd48c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4800503e0 con 0x7fd48c0b1730 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.356+0000 7fd4897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:12.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.597+0000 7fd492a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.597+0000 7fd492a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.597+0000 7fd492a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:12.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.597+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd48c145010 con 0x7fd48c0b1730 2024-03-25T19:36:12.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.632+0000 7fd4897fa640 1 -- 172.21.15.29:0/2587147526 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fd480052310 con 0x7fd48c0b1730 2024-03-25T19:36:12.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.632+0000 7fd4897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.632+0000 7fd4897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:12.631 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 10 librados: watch_flush enter 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 10 librados: watch_flush exit 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 >> v1:172.21.15.29:6800/2314944989 conn(0x7fd47006df70 legacy=0x7fd470070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 10 monclient: shutdown 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 >> v1:172.21.15.29:6789/0 conn(0x7fd48c0b1730 legacy=0x7fd48c140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 shutdown_connections 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 >> 172.21.15.29:0/2587147526 conn(0x7fd48c0617a0 msgr2=0x7fd48c13c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 shutdown_connections 2024-03-25T19:36:12.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 -- 172.21.15.29:0/2587147526 wait complete. 2024-03-25T19:36:12.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:12.634+0000 7fd492a7b640 1 librados: shutdown 2024-03-25T19:36:12.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:19.453 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:19.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.572+0000 7f46fa7e2640 10 monclient: get_monmap_and_config 2024-03-25T19:36:19.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.572+0000 7f46fa7e2640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.572+0000 7f46fa7e2640 10 monclient: monmap: 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:19.573966+0000 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:19.573966+0000 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:19.573 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.573+0000 7f46fa7e2640 1 Processor -- start 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.573+0000 7f46fa7e2640 1 -- start start 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.573+0000 7f46fa7e2640 10 monclient: init 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f46f40a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f46f413f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f46f41430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f4142520 con 0x7f46f41430c0 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f40b28a0 con 0x7f46f413f710 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f40b2a30 con 0x7f46f40a98f0 2024-03-25T19:36:19.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.574+0000 7f46fa7e2640 10 monclient(hunting): authenticate will time out at 6256.328163s 2024-03-25T19:36:19.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.585+0000 7f46f8d7f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46f41430c0 0x7f46f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46784/0 (socket says 172.21.15.29:46784) 2024-03-25T19:36:19.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.585+0000 7f46f8d7f640 1 -- 172.21.15.29:0/1131844042 learned_addr learned my addr 172.21.15.29:0/1131844042 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353462489 0 0) 0x7f46f4142520 con 0x7f46f41430c0 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:36:19.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.603+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc003520 con 0x7f46f41430c0 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777249515 0 0) 0x7f46f40b28a0 con 0x7f46f413f710 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:36:19.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.607+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4142520 con 0x7f46f413f710 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506065959 0 0) 0x7f46f40b2a30 con 0x7f46f40a98f0 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:36:19.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.615+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f40b28a0 con 0x7f46f40a98f0 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.623+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233390645 0 0) 0x7f46f40b28a0 con 0x7f46f40a98f0 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.623+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 >> v1:172.21.15.196:6789/0 conn(0x7f46f413f710 legacy=0x7f46f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 >> v1:172.21.15.29:6789/0 conn(0x7f46f41430c0 legacy=0x7f46f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f40b44c0 con 0x7f46f40a98f0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f46e40015b0 con 0x7f46f40a98f0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46fa7e2640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46fa7e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46f2ffd640 10 monclient: dump: 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:19.627 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:19.627 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.624+0000 7f46fa7e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:19.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46e40017d0 con 0x7f46f40a98f0 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 1 -- 172.21.15.29:0/1131844042 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f46e4001b60 con 0x7f46f40a98f0 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46f2ffd640 10 monclient: dump: 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:19.661 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.661 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46fa7e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:19.661 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.659+0000 7f46fa7e2640 4 set_mon_vals no callback set 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 10 monclient: shutdown 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1131844042 >> v1:172.21.15.29:6790/0 conn(0x7f46f40a98f0 legacy=0x7f46f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1131844042 shutdown_connections 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1131844042 >> 172.21.15.29:0/1131844042 conn(0x7f46f40b7ea0 msgr2=0x7f46f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1131844042 shutdown_connections 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1131844042 wait complete. 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.660+0000 7f46fa7e2640 10 monclient: monmap: 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 librados: starting msgr at 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 librados: starting objecter 2024-03-25T19:36:19.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 Processor -- start 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 -- start start 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 librados: setting wanted keys 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 1 librados: calling monclient init 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.661+0000 7f46fa7e2640 10 monclient: init 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f46f40a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f46f413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f46f41430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f41d51f0 con 0x7f46f413f710 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f4138c70 con 0x7f46f41430c0 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f4138e00 con 0x7f46f40a98f0 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.662+0000 7f46fa7e2640 10 monclient(hunting): authenticate will time out at 6256.416200s 2024-03-25T19:36:19.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.673+0000 7f46f37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46f413f710 0x7f46f41d7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46856/0 (socket says 172.21.15.29:46856) 2024-03-25T19:36:19.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.673+0000 7f46f37fe640 1 -- 172.21.15.29:0/1113960063 learned_addr learned my addr 172.21.15.29:0/1113960063 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:19.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.678+0000 7f46f0ff9640 10 client.?.objecter ms_handle_connect 0x7f46f41430c0 2024-03-25T19:36:19.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.678+0000 7f46f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.679+0000 7f46f0ff9640 10 client.?.objecter ms_handle_connect 0x7f46f413f710 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.679+0000 7f46f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.681+0000 7f46f0ff9640 10 client.?.objecter ms_handle_connect 0x7f46f40a98f0 2024-03-25T19:36:19.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.681+0000 7f46f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844970825 0 0) 0x7f46f4138c70 con 0x7f46f41430c0 2024-03-25T19:36:19.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:36:19.687 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.686+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e0004eb0 con 0x7f46f41430c0 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805798025 0 0) 0x7f46f4138e00 con 0x7f46f40a98f0 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:36:19.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.690+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4138c70 con 0x7f46f40a98f0 2024-03-25T19:36:19.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641298780 0 0) 0x7f46e0004eb0 con 0x7f46f41430c0 2024-03-25T19:36:19.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f4138e00 con 0x7f46f41430c0 2024-03-25T19:36:19.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f46ec001b30 con 0x7f46f41430c0 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.708+0000 7f46f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342010874 0 0) 0x7f46f41d51f0 con 0x7f46f413f710 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:36:19.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.712+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e0004eb0 con 0x7f46f413f710 2024-03-25T19:36:19.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517673989 0 0) 0x7f46f4138c70 con 0x7f46f40a98f0 2024-03-25T19:36:19.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f41d51f0 con 0x7f46f40a98f0 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f46e40014f0 con 0x7f46f40a98f0 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.714+0000 7f46f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:19.715 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2207743357 0 0) 0x7f46f4138e00 con 0x7f46f41430c0 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 >> v1:172.21.15.29:6790/0 conn(0x7f46f40a98f0 legacy=0x7f46f4138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 >> v1:172.21.15.29:6789/0 conn(0x7f46f413f710 legacy=0x7f46f41d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 1 monclient: found mon.b 2024-03-25T19:36:19.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f41d8390 con 0x7f46f41430c0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 10 monclient: _renew_subs 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f4137850 con 0x7f46f41430c0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 10 monclient: _renew_subs 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f41e3550 con 0x7f46f41430c0 2024-03-25T19:36:19.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.729+0000 7f46fa7e2640 1 librados: init done 2024-03-25T19:36:19.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.730+0000 7f46fa7e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.730+0000 7f46fa7e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.730+0000 7f46fa7e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:19.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:19.730+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f40abbe0 con 0x7f46f41430c0 2024-03-25T19:36:20.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46ec002020 con 0x7f46f41430c0 2024-03-25T19:36:20.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f46ec004d00 con 0x7f46f41430c0 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f0ff9640 10 monclient: dump: 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:20.010 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:20.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.009+0000 7f46f27fc640 4 set_mon_vals no callback set 2024-03-25T19:36:20.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.010+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f46ec01acd0 con 0x7f46f41430c0 2024-03-25T19:36:20.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.010+0000 7f46f3fff640 10 monclient: get_auth_request con 0x7f46e006df70 auth_method 0 2024-03-25T19:36:20.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.010+0000 7f46f0ff9640 10 client.4330.objecter ms_handle_connect 0x7f46e006df70 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f46ec0549a0 con 0x7f46f41430c0 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 10 client.4330.objecter ms_dispatch 0x7f46f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46ec0502e0 con 0x7f46f41430c0 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.015+0000 7f46f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:20.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.272+0000 7f46fa7e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.272+0000 7f46fa7e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.272+0000 7f46fa7e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:20.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.272+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46f4074cc0 con 0x7f46f41430c0 2024-03-25T19:36:20.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.296+0000 7f46f0ff9640 1 -- 172.21.15.29:0/1113960063 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f46ec0523e0 con 0x7f46f41430c0 2024-03-25T19:36:20.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.296+0000 7f46f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.296+0000 7f46f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:20.297 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:20.297 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:20.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 10 librados: watch_flush enter 2024-03-25T19:36:20.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 10 librados: watch_flush exit 2024-03-25T19:36:20.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T19:36:20.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 10 client.4330.objecter successfully canceled tick 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 >> v1:172.21.15.29:6800/2314944989 conn(0x7f46e006df70 legacy=0x7f46e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 10 monclient: shutdown 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 >> v1:172.21.15.196:6789/0 conn(0x7f46f41430c0 legacy=0x7f46f41e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 shutdown_connections 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 >> 172.21.15.29:0/1113960063 conn(0x7f46f40b7ea0 msgr2=0x7f46f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 shutdown_connections 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 -- 172.21.15.29:0/1113960063 wait complete. 2024-03-25T19:36:20.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:20.298+0000 7f46fa7e2640 1 librados: shutdown 2024-03-25T19:36:20.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:24.634 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:54.797 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.652 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:54.816 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.692 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:54.856 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.697 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:54.861 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.709 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:54.871 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.719 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:54.882 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.721 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:54.884 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.742 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:02:54.904 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.744 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:54.908 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:27.118 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:27.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.230+0000 7f1b22172640 10 monclient: get_monmap_and_config 2024-03-25T19:36:27.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.230+0000 7f1b22172640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.230+0000 7f1b22172640 10 monclient: monmap: 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:27.232179+0000 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:27.232179+0000 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:27.231 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.231+0000 7f1b22172640 1 Processor -- start 2024-03-25T19:36:27.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.231+0000 7f1b22172640 1 -- start start 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.231+0000 7f1b22172640 10 monclient: init 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-c con 0x7f1b1c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-a con 0x7f1b1c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c142520 con 0x7f1b1c13f710 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0b28a0 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0b2a30 con 0x7f1b1c1430c0 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.232+0000 7f1b22172640 10 monclient(hunting): authenticate will time out at 6263.986321s 2024-03-25T19:36:27.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.238+0000 7f1b1affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1b1c13f710 0x7f1b1c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51182/0 (socket says 172.21.15.29:51182) 2024-03-25T19:36:27.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.238+0000 7f1b1affd640 1 -- 172.21.15.29:0/1853288903 learned_addr learned my addr 172.21.15.29:0/1853288903 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:27.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741752711 0 0) 0x7f1b1c142520 con 0x7f1b1c13f710 2024-03-25T19:36:27.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:36:27.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.249+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003520 con 0x7f1b1c13f710 2024-03-25T19:36:27.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977206235 0 0) 0x7f1b04003520 con 0x7f1b1c13f710 2024-03-25T19:36:27.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 >> v1:172.21.15.29:6790/0 conn(0x7f1b1c1430c0 legacy=0x7f1b1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:27.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 >> v1:172.21.15.196:6789/0 conn(0x7f1b1c0a98f0 legacy=0x7f1b1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c0b44c0 con 0x7f1b1c13f710 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b10001690 con 0x7f1b1c13f710 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b22172640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b22172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b1a7fc640 10 monclient: dump: 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.260 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.258+0000 7f1b22172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b10001860 con 0x7f1b1c13f710 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 1 -- 172.21.15.29:0/1853288903 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b10001cb0 con 0x7f1b1c13f710 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b22172640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b22172640 4 set_mon_vals no callback set 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.262+0000 7f1b1a7fc640 10 monclient: dump: 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.263 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.264 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 10 monclient: shutdown 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 1 -- 172.21.15.29:0/1853288903 >> v1:172.21.15.29:6789/0 conn(0x7f1b1c13f710 legacy=0x7f1b1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 1 -- 172.21.15.29:0/1853288903 shutdown_connections 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 1 -- 172.21.15.29:0/1853288903 >> 172.21.15.29:0/1853288903 conn(0x7f1b1c0b7ea0 msgr2=0x7f1b1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 1 -- 172.21.15.29:0/1853288903 shutdown_connections 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.263+0000 7f1b22172640 1 -- 172.21.15.29:0/1853288903 wait complete. 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.264+0000 7f1b22172640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.264+0000 7f1b22172640 10 monclient: monmap: 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 librados: starting msgr at 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 librados: starting objecter 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 Processor -- start 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 -- start start 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 librados: setting wanted keys 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 1 librados: calling monclient init 2024-03-25T19:36:27.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.265+0000 7f1b22172640 10 monclient: init 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-c con 0x7f1b1c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient(hunting): picked mon.noname-a con 0x7f1b1c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0b0bc0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0afc80 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c14e5b0 con 0x7f1b1c13f710 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.266+0000 7f1b22172640 10 monclient(hunting): authenticate will time out at 6264.020147s 2024-03-25T19:36:27.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.272+0000 7f1b1bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1b1c1430c0 0x7f1b1c1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51216/0 (socket says 172.21.15.29:51216) 2024-03-25T19:36:27.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.272+0000 7f1b1bfff640 1 -- 172.21.15.29:0/2347887203 learned_addr learned my addr 172.21.15.29:0/2347887203 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:27.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.273+0000 7f1b03fff640 10 client.?.objecter ms_handle_connect 0x7f1b1c1430c0 2024-03-25T19:36:27.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.273+0000 7f1b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.278+0000 7f1b03fff640 10 client.?.objecter ms_handle_connect 0x7f1b1c0a98f0 2024-03-25T19:36:27.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.278+0000 7f1b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.282+0000 7f1b03fff640 10 client.?.objecter ms_handle_connect 0x7f1b1c13f710 2024-03-25T19:36:27.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.282+0000 7f1b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201179266 0 0) 0x7f1b1c0b0bc0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:36:27.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.289+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b08004eb0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302155570 0 0) 0x7f1b1c14e5b0 con 0x7f1b1c13f710 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:36:27.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.296+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c0b0bc0 con 0x7f1b1c13f710 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792984135 0 0) 0x7f1b1c0afc80 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:36:27.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.299+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c14e5b0 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335308994 0 0) 0x7f1b1c14e5b0 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b1c0afc80 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b0c001830 con 0x7f1b1c0a98f0 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 10 monclient(hunting): dump: 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.312 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.313 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.313 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.311+0000 7f1b03fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901278786 0 0) 0x7f1b08004eb0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b1c14e5b0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b14001ad0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.315+0000 7f1b03fff640 10 monclient(hunting): dump: 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.316 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530944293 0 0) 0x7f1b1c0b0bc0 con 0x7f1b1c13f710 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b08004eb0 con 0x7f1b1c13f710 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b10001b70 con 0x7f1b1c13f710 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.325+0000 7f1b03fff640 10 monclient(hunting): dump: 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.326 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.327 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.327 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.327 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2524589008 0 0) 0x7f1b1c14e5b0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 >> v1:172.21.15.29:6790/0 conn(0x7f1b1c13f710 legacy=0x7f1b1c0ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 >> v1:172.21.15.196:6789/0 conn(0x7f1b1c0a98f0 legacy=0x7f1b1c0af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 1 monclient: found mon.a 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c14e740 con 0x7f1b1c1430c0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 10 monclient: _renew_subs 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c0b1230 con 0x7f1b1c1430c0 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:27.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 10 monclient: _renew_subs 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.340+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1c1e3550 con 0x7f1b1c1430c0 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.341+0000 7f1b22172640 1 librados: init done 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.342+0000 7f1b22172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.342+0000 7f1b22172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.342+0000 7f1b22172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.342+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c0af9a0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b14001fe0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b14004be0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b03fff640 10 monclient: dump: 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.693+0000 7f1b19ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:27.696 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.694+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f1b1401abb0 con 0x7f1b1c1430c0 2024-03-25T19:36:27.696 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.695+0000 7f1b1b7fe640 10 monclient: get_auth_request con 0x7f1b0806e0c0 auth_method 0 2024-03-25T19:36:27.696 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.695+0000 7f1b03fff640 10 client.4335.objecter ms_handle_connect 0x7f1b0806e0c0 2024-03-25T19:36:27.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f1b14054850 con 0x7f1b1c1430c0 2024-03-25T19:36:27.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 10 client.4335.objecter ms_dispatch 0x7f1b1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:27.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:27.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b14050360 con 0x7f1b1c1430c0 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.701+0000 7f1b03fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:27.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.956+0000 7f1b22172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.956+0000 7f1b22172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.956+0000 7f1b22172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:27.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:27.956+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b1c0627b0 con 0x7f1b1c1430c0 2024-03-25T19:36:28.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.016+0000 7f1b03fff640 1 -- 172.21.15.29:0/2347887203 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f1b14052290 con 0x7f1b1c1430c0 2024-03-25T19:36:28.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.016+0000 7f1b03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.016+0000 7f1b03fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:28.017 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:28.017 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:28.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 10 librados: watch_flush enter 2024-03-25T19:36:28.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 10 librados: watch_flush exit 2024-03-25T19:36:28.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:36:28.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:36:28.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 >> v1:172.21.15.29:6800/2314944989 conn(0x7f1b0806e0c0 legacy=0x7f1b08070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 10 monclient: shutdown 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 >> v1:172.21.15.29:6789/0 conn(0x7f1b1c1430c0 legacy=0x7f1b1c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 shutdown_connections 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 >> 172.21.15.29:0/2347887203 conn(0x7f1b1c0b7ea0 msgr2=0x7f1b1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 shutdown_connections 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.019+0000 7f1b22172640 1 -- 172.21.15.29:0/2347887203 wait complete. 2024-03-25T19:36:28.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:28.020+0000 7f1b22172640 1 librados: shutdown 2024-03-25T19:36:28.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:34.837 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.954+0000 7f5bd8dcf640 10 monclient: get_monmap_and_config 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.954+0000 7f5bd8dcf640 10 monclient: build_initial_monmap 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.954+0000 7f5bd8dcf640 10 monclient: monmap: 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:34.955497+0000 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:34.955497+0000 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:34.954 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:34.955 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:34.955 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:34.955 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:34.955 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.955+0000 7f5bd8dcf640 1 Processor -- start 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.955+0000 7f5bd8dcf640 1 -- start start 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.955+0000 7f5bd8dcf640 10 monclient: init 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd40a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd41430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4142520 con 0x7f5bd41430c0 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b28a0 con 0x7f5bd40a98f0 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b2a30 con 0x7f5bd413f710 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.956+0000 7f5bd8dcf640 10 monclient(hunting): authenticate will time out at 6271.709381s 2024-03-25T19:36:34.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.969+0000 7f5bd259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5bd40a98f0 0x7f5bd40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:45608/0 (socket says 172.21.15.29:45608) 2024-03-25T19:36:34.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.969+0000 7f5bd259c640 1 -- 172.21.15.29:0/1425690921 learned_addr learned my addr 172.21.15.29:0/1425690921 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281361189 0 0) 0x7f5bd4142520 con 0x7f5bd41430c0 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:36:34.988 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.988+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc003520 con 0x7f5bd41430c0 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937859427 0 0) 0x7f5bd40b28a0 con 0x7f5bd40a98f0 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:36:34.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.993+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd4142520 con 0x7f5bd40a98f0 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091875199 0 0) 0x7f5bd40b2a30 con 0x7f5bd413f710 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:36:34.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:34.998+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40b28a0 con 0x7f5bd413f710 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659868084 0 0) 0x7f5bbc003520 con 0x7f5bd41430c0 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 >> v1:172.21.15.29:6790/0 conn(0x7f5bd413f710 legacy=0x7f5bd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 >> v1:172.21.15.196:6789/0 conn(0x7f5bd40a98f0 legacy=0x7f5bd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 monclient: found mon.noname-a 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40b44c0 con 0x7f5bd41430c0 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5bcc001a10 con 0x7f5bd41430c0 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd8dcf640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd8dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd159a640 10 monclient: dump: 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:35.005 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:35.006 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:35.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.004+0000 7f5bd8dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bcc001b70 con 0x7f5bd41430c0 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 1 -- 172.21.15.29:0/1425690921 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5bcc001f80 con 0x7f5bd41430c0 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 4 set_mon_vals no callback set 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:35.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd159a640 10 monclient: dump: 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:35.019 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:35.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 10 monclient: shutdown 2024-03-25T19:36:35.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/1425690921 >> v1:172.21.15.29:6789/0 conn(0x7f5bd41430c0 legacy=0x7f5bd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/1425690921 shutdown_connections 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/1425690921 >> 172.21.15.29:0/1425690921 conn(0x7f5bd40b7ea0 msgr2=0x7f5bd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/1425690921 shutdown_connections 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.019+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/1425690921 wait complete. 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.020+0000 7f5bd8dcf640 10 monclient: build_initial_monmap 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.020+0000 7f5bd8dcf640 10 monclient: monmap: 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:35.022 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 librados: starting msgr at 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 librados: starting objecter 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 Processor -- start 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 -- start start 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 librados: setting wanted keys 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 1 librados: calling monclient init 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.021+0000 7f5bd8dcf640 10 monclient: init 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd40a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd41430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4136a50 con 0x7f5bd413f710 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b79c0 con 0x7f5bd41430c0 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b7ca0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:35.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.022+0000 7f5bd8dcf640 10 monclient(hunting): authenticate will time out at 6271.774990s 2024-03-25T19:36:35.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.038+0000 7f5bd259c640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f5bd40a98f0 0x7f5bd4137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:43238/0 (socket says 172.21.15.29:43238) 2024-03-25T19:36:35.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.038+0000 7f5bd259c640 1 -- 172.21.15.29:0/813163648 learned_addr learned my addr 172.21.15.29:0/813163648 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:35.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.039+0000 7f5bc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5bd40a98f0 2024-03-25T19:36:35.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.039+0000 7f5bc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.043+0000 7f5bc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5bd41430c0 2024-03-25T19:36:35.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.043+0000 7f5bc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.047+0000 7f5bc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5bd413f710 2024-03-25T19:36:35.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.047+0000 7f5bc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897420079 0 0) 0x7f5bd40b7ca0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.055 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.055 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:36:35.055 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.055+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb4004eb0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945247491 0 0) 0x7f5bd40b79c0 con 0x7f5bd41430c0 2024-03-25T19:36:35.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:36:35.058 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.058+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40b7ca0 con 0x7f5bd41430c0 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534790646 0 0) 0x7f5bd4136a50 con 0x7f5bd413f710 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:36:35.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.065+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40b79c0 con 0x7f5bd413f710 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355426809 0 0) 0x7f5bb4004eb0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd4136a50 con 0x7f5bd40a98f0 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5bc40019f0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.068 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:35.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.068+0000 7f5bc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:35.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3495573891 0 0) 0x7f5bd4136a50 con 0x7f5bd40a98f0 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 >> v1:172.21.15.196:6789/0 conn(0x7f5bd41430c0 legacy=0x7f5bd41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 >> v1:172.21.15.29:6789/0 conn(0x7f5bd413f710 legacy=0x7f5bd4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 1 monclient: found mon.c 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd4136ca0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 10 monclient: _renew_subs 2024-03-25T19:36:35.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd4136f80 con 0x7f5bd40a98f0 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 10 monclient: _renew_subs 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd41360e0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.074+0000 7f5bd8dcf640 1 librados: init done 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.075+0000 7f5bd8dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.075+0000 7f5bd8dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.075+0000 7f5bd8dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:35.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.075+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd40627b0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc4001cb0 con 0x7f5bd40a98f0 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5bc4004c30 con 0x7f5bd40a98f0 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bc2ffd640 10 monclient: dump: 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.079 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:35.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.079+0000 7f5bd0d99640 4 set_mon_vals no callback set 2024-03-25T19:36:35.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.398+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f5bc401ac00 con 0x7f5bd40a98f0 2024-03-25T19:36:35.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.399+0000 7f5bd1d9b640 10 monclient: get_auth_request con 0x7f5bb406dd50 auth_method 0 2024-03-25T19:36:35.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.399+0000 7f5bc2ffd640 10 client.4340.objecter ms_handle_connect 0x7f5bb406dd50 2024-03-25T19:36:35.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f5bc4054930 con 0x7f5bd40a98f0 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 10 client.4340.objecter ms_dispatch 0x7f5bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bc4050270 con 0x7f5bd40a98f0 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.406+0000 7f5bc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:35.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.648+0000 7f5bd8dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.648+0000 7f5bd8dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.648+0000 7f5bd8dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:35.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.648+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bd413e290 con 0x7f5bd40a98f0 2024-03-25T19:36:35.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.769+0000 7f5bc2ffd640 1 -- 172.21.15.29:0/813163648 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f5bc4052370 con 0x7f5bd40a98f0 2024-03-25T19:36:35.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.769+0000 7f5bc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.769+0000 7f5bc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:35.769 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:35.769 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 10 librados: watch_flush enter 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 10 librados: watch_flush exit 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 10 client.4340.objecter successfully canceled tick 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 >> v1:172.21.15.29:6800/2314944989 conn(0x7f5bb406dd50 legacy=0x7f5bb4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 10 monclient: shutdown 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 >> v1:172.21.15.29:6790/0 conn(0x7f5bd40a98f0 legacy=0x7f5bd4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 shutdown_connections 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 >> 172.21.15.29:0/813163648 conn(0x7f5bd40b7ea0 msgr2=0x7f5bd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 shutdown_connections 2024-03-25T19:36:35.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 -- 172.21.15.29:0/813163648 wait complete. 2024-03-25T19:36:35.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:35.772+0000 7f5bd8dcf640 1 librados: shutdown 2024-03-25T19:36:35.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:42.590 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:42.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.703+0000 7f1bcaa7b640 10 monclient: get_monmap_and_config 2024-03-25T19:36:42.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.703+0000 7f1bcaa7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.703+0000 7f1bcaa7b640 10 monclient: monmap: 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:42.704269+0000 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:42.704269+0000 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:42.703 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.704+0000 7f1bcaa7b640 1 Processor -- start 2024-03-25T19:36:42.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.704+0000 7f1bcaa7b640 1 -- start start 2024-03-25T19:36:42.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.704+0000 7f1bcaa7b640 10 monclient: init 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc40705b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc41402d0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc4149ae0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40718c0 con 0x7f1bc41402d0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40714c0 con 0x7f1bc40705b0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc4072b50 con 0x7f1bc4149ae0 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.705+0000 7f1bcaa7b640 10 monclient(hunting): authenticate will time out at 6279.458522s 2024-03-25T19:36:42.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.710+0000 7f1bc3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1bc41402d0 0x7f1bc4140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59558/0 (socket says 172.21.15.29:59558) 2024-03-25T19:36:42.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.710+0000 7f1bc3fff640 1 -- 172.21.15.29:0/2280527089 learned_addr learned my addr 172.21.15.29:0/2280527089 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.718+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911613041 0 0) 0x7f1bc40718c0 con 0x7f1bc41402d0 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.719+0000 7f1bc37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.719+0000 7f1bc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.719+0000 7f1bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.719+0000 7f1bc37fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:36:42.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.719+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb4003520 con 0x7f1bc41402d0 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37534639 0 0) 0x7f1bc40714c0 con 0x7f1bc40705b0 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:36:42.730 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.731+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc40718c0 con 0x7f1bc40705b0 2024-03-25T19:36:42.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339476447 0 0) 0x7f1bb4003520 con 0x7f1bc41402d0 2024-03-25T19:36:42.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 >> v1:172.21.15.29:6790/0 conn(0x7f1bc4149ae0 legacy=0x7f1bc414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 >> v1:172.21.15.196:6789/0 conn(0x7f1bc40705b0 legacy=0x7f1bc413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 monclient: found mon.noname-a 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc413dd10 con 0x7f1bc41402d0 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1bb0001690 con 0x7f1bc41402d0 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bc37fe640 10 monclient: dump: 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.736 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bcaa7b640 5 monclient: authenticate success, global_id 4344 2024-03-25T19:36:42.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.736+0000 7f1bcaa7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:42.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb00017f0 con 0x7f1bc41402d0 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 1 -- 172.21.15.29:0/2280527089 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1bb0001c00 con 0x7f1bc41402d0 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bcaa7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bcaa7b640 4 set_mon_vals no callback set 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.746+0000 7f1bc37fe640 10 monclient: dump: 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:42.746 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:42.747 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:42.747 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 10 monclient: shutdown 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/2280527089 >> v1:172.21.15.29:6789/0 conn(0x7f1bc41402d0 legacy=0x7f1bc4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/2280527089 shutdown_connections 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/2280527089 >> 172.21.15.29:0/2280527089 conn(0x7f1bc40b5d10 msgr2=0x7f1bc40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/2280527089 shutdown_connections 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.747+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/2280527089 wait complete. 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 10 monclient: monmap: 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:42.749 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 librados: starting msgr at 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 librados: starting objecter 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 Processor -- start 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 -- start start 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 librados: setting wanted keys 2024-03-25T19:36:42.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 1 librados: calling monclient init 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.748+0000 7f1bcaa7b640 10 monclient: init 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc40705b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc41402d0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc4149ae0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc41433d0 con 0x7f1bc40705b0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc413ace0 con 0x7f1bc4149ae0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc4144050 con 0x7f1bc41402d0 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.751 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.749+0000 7f1bcaa7b640 10 monclient(hunting): authenticate will time out at 6279.502660s 2024-03-25T19:36:42.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.756+0000 7f1bc9018640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1bc4149ae0 0x7f1bc41ddea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:43928/0 (socket says 172.21.15.29:43928) 2024-03-25T19:36:42.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.756+0000 7f1bc9018640 1 -- 172.21.15.29:0/1795135117 learned_addr learned my addr 172.21.15.29:0/1795135117 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:42.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.760+0000 7f1bc17fa640 10 client.?.objecter ms_handle_connect 0x7f1bc4149ae0 2024-03-25T19:36:42.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.760+0000 7f1bc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.761+0000 7f1bc17fa640 10 client.?.objecter ms_handle_connect 0x7f1bc41402d0 2024-03-25T19:36:42.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.761+0000 7f1bc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193423675 0 0) 0x7f1bc4072b50 con 0x7f1bc4149ae0 2024-03-25T19:36:42.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:36:42.764 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.764+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba8004eb0 con 0x7f1bc4149ae0 2024-03-25T19:36:42.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.768+0000 7f1bc17fa640 10 client.?.objecter ms_handle_connect 0x7f1bc40705b0 2024-03-25T19:36:42.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.768+0000 7f1bc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.770 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091737115 0 0) 0x7f1bc4144050 con 0x7f1bc41402d0 2024-03-25T19:36:42.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:36:42.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.771+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4072b50 con 0x7f1bc41402d0 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3332061903 0 0) 0x7f1ba8004eb0 con 0x7f1bc4149ae0 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc4144050 con 0x7f1bc4149ae0 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1bbc001a90 con 0x7f1bc4149ae0 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 10 monclient(hunting): dump: 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:42.780 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.780+0000 7f1bc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:42.782 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165452210 0 0) 0x7f1bc4072b50 con 0x7f1bc41402d0 2024-03-25T19:36:42.782 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ba8004eb0 con 0x7f1bc41402d0 2024-03-25T19:36:42.782 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1bb0001b00 con 0x7f1bc41402d0 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.783+0000 7f1bc17fa640 10 monclient(hunting): dump: 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:42.783 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:42.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194408808 0 0) 0x7f1bc41433d0 con 0x7f1bc40705b0 2024-03-25T19:36:42.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:36:42.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.789+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4072b50 con 0x7f1bc40705b0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040581572 0 0) 0x7f1ba8004eb0 con 0x7f1bc41402d0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 >> v1:172.21.15.196:6789/0 conn(0x7f1bc4149ae0 legacy=0x7f1bc41ddea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 >> v1:172.21.15.29:6789/0 conn(0x7f1bc40705b0 legacy=0x7f1bc4142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 1 monclient: found mon.c 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc4137f20 con 0x7f1bc41402d0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 10 monclient: _renew_subs 2024-03-25T19:36:42.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc41442a0 con 0x7f1bc41402d0 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 10 monclient: _renew_subs 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc4142200 con 0x7f1bc41402d0 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.796+0000 7f1bcaa7b640 1 librados: init done 2024-03-25T19:36:42.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.797+0000 7f1bcaa7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.797+0000 7f1bcaa7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.797+0000 7f1bcaa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:42.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:42.797+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc4143700 con 0x7f1bc41402d0 2024-03-25T19:36:43.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb0004a90 con 0x7f1bc41402d0 2024-03-25T19:36:43.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:43.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1bb0001cc0 con 0x7f1bc41402d0 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 10 monclient: dump: 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:43.123 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:43.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc2ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:43.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.123+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f1bb001ac10 con 0x7f1bc41402d0 2024-03-25T19:36:43.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.124+0000 7f1bc8817640 10 monclient: get_auth_request con 0x7f1ba806df30 auth_method 0 2024-03-25T19:36:43.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.124+0000 7f1bc17fa640 10 client.4346.objecter ms_handle_connect 0x7f1ba806df30 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f1bb0054810 con 0x7f1bc41402d0 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 10 client.4346.objecter ms_dispatch 0x7f1bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bb0050150 con 0x7f1bc41402d0 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.131+0000 7f1bc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:43.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.399+0000 7f1bcaa7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.399+0000 7f1bcaa7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.399+0000 7f1bcaa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:36:43.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.399+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc4141440 con 0x7f1bc41402d0 2024-03-25T19:36:43.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.436+0000 7f1bc17fa640 1 -- 172.21.15.29:0/1795135117 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f1bb0052250 con 0x7f1bc41402d0 2024-03-25T19:36:43.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.436+0000 7f1bc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.436+0000 7f1bc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:43.436 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:43.436 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 10 librados: watch_flush enter 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 10 librados: watch_flush exit 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 10 client.4346.objecter successfully canceled tick 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 >> v1:172.21.15.29:6800/2314944989 conn(0x7f1ba806df30 legacy=0x7f1ba80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 10 monclient: shutdown 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 >> v1:172.21.15.29:6790/0 conn(0x7f1bc41402d0 legacy=0x7f1bc4137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 shutdown_connections 2024-03-25T19:36:43.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 >> 172.21.15.29:0/1795135117 conn(0x7f1bc40b5d10 msgr2=0x7f1bc40b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 shutdown_connections 2024-03-25T19:36:43.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 -- 172.21.15.29:0/1795135117 wait complete. 2024-03-25T19:36:43.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:43.439+0000 7f1bcaa7b640 1 librados: shutdown 2024-03-25T19:36:43.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:44.708 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:14.872 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.763 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:14.926 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.786 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:14.950 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.827 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:14.991 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.839 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:15.001 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.839 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:15.003 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.845 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:15.007 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.868 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:15.032 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.880 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:15.042 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:50.256 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:50.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.371+0000 7fc91a915640 10 monclient: get_monmap_and_config 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.371+0000 7fc91a915640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.371+0000 7fc91a915640 10 monclient: monmap: 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:50.372183+0000 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:50.372183+0000 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:50.371 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.372+0000 7fc91a915640 1 Processor -- start 2024-03-25T19:36:50.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.372+0000 7fc91a915640 1 -- start start 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.372+0000 7fc91a915640 10 monclient: init 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-b con 0x7fc9140a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-a con 0x7fc91413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-c con 0x7fc9141430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914142520 con 0x7fc91413f710 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140b28a0 con 0x7fc9141430c0 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140b2a30 con 0x7fc9140a98f0 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.373+0000 7fc91a915640 10 monclient(hunting): authenticate will time out at 6287.126082s 2024-03-25T19:36:50.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.387+0000 7fc918eb2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc9141430c0 0x7fc91414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41324/0 (socket says 172.21.15.29:41324) 2024-03-25T19:36:50.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.387+0000 7fc918eb2640 1 -- 172.21.15.29:0/1976081635 learned_addr learned my addr 172.21.15.29:0/1976081635 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:50.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022791251 0 0) 0x7fc9140b28a0 con 0x7fc9141430c0 2024-03-25T19:36:50.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:36:50.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.401+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900003520 con 0x7fc9141430c0 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306310854 0 0) 0x7fc9140b2a30 con 0x7fc9140a98f0 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:36:50.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.410+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9140b28a0 con 0x7fc9140a98f0 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280339936 0 0) 0x7fc914142520 con 0x7fc91413f710 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:36:50.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.413+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9140b2a30 con 0x7fc91413f710 2024-03-25T19:36:50.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635244863 0 0) 0x7fc900003520 con 0x7fc9141430c0 2024-03-25T19:36:50.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 >> v1:172.21.15.29:6790/0 conn(0x7fc9140a98f0 legacy=0x7fc9140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 >> v1:172.21.15.29:6789/0 conn(0x7fc91413f710 legacy=0x7fc9141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 monclient: found mon.noname-c 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140b44c0 con 0x7fc9141430c0 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc904001690 con 0x7fc9141430c0 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc912ffd640 10 monclient: dump: 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.425 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc91a915640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:36:50.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.424+0000 7fc91a915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9040017f0 con 0x7fc9141430c0 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 1 -- 172.21.15.29:0/1976081635 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc904001c00 con 0x7fc9141430c0 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc91a915640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc91a915640 4 set_mon_vals no callback set 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.438+0000 7fc912ffd640 10 monclient: dump: 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.438 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.439 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.439 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.439 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.439 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.439 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 10 monclient: shutdown 2024-03-25T19:36:50.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 1 -- 172.21.15.29:0/1976081635 >> v1:172.21.15.196:6789/0 conn(0x7fc9141430c0 legacy=0x7fc91414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 1 -- 172.21.15.29:0/1976081635 shutdown_connections 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 1 -- 172.21.15.29:0/1976081635 >> 172.21.15.29:0/1976081635 conn(0x7fc9140b7ea0 msgr2=0x7fc9140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 1 -- 172.21.15.29:0/1976081635 shutdown_connections 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.439+0000 7fc91a915640 1 -- 172.21.15.29:0/1976081635 wait complete. 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.440+0000 7fc91a915640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.440+0000 7fc91a915640 10 monclient: monmap: 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:50.441 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.440+0000 7fc91a915640 1 librados: starting msgr at 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.440+0000 7fc91a915640 1 librados: starting objecter 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.441+0000 7fc91a915640 1 Processor -- start 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.441+0000 7fc91a915640 1 -- start start 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.441+0000 7fc91a915640 1 librados: setting wanted keys 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.441+0000 7fc91a915640 1 librados: calling monclient init 2024-03-25T19:36:50.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.441+0000 7fc91a915640 10 monclient: init 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-a con 0x7fc9140a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-c con 0x7fc91413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient(hunting): picked mon.noname-b con 0x7fc9141430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914135230 con 0x7fc9140a98f0 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914135da0 con 0x7fc9141430c0 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914135f30 con 0x7fc91413f710 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.442+0000 7fc91a915640 10 monclient(hunting): authenticate will time out at 6287.194955s 2024-03-25T19:36:50.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.449+0000 7fc913fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc9140a98f0 0x7fc9141d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35656/0 (socket says 172.21.15.29:35656) 2024-03-25T19:36:50.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.449+0000 7fc913fff640 1 -- 172.21.15.29:0/4170004864 learned_addr learned my addr 172.21.15.29:0/4170004864 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:50.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.454+0000 7fc910ff9640 10 client.?.objecter ms_handle_connect 0x7fc9140a98f0 2024-03-25T19:36:50.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.454+0000 7fc910ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.463+0000 7fc910ff9640 10 client.?.objecter ms_handle_connect 0x7fc91413f710 2024-03-25T19:36:50.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.463+0000 7fc910ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.468+0000 7fc910ff9640 10 client.?.objecter ms_handle_connect 0x7fc9141430c0 2024-03-25T19:36:50.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.468+0000 7fc910ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866342977 0 0) 0x7fc914135230 con 0x7fc9140a98f0 2024-03-25T19:36:50.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:36:50.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.470+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4004eb0 con 0x7fc9140a98f0 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888495812 0 0) 0x7fc914135f30 con 0x7fc91413f710 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:36:50.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.475+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914135230 con 0x7fc91413f710 2024-03-25T19:36:50.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061699525 0 0) 0x7fc914135da0 con 0x7fc9141430c0 2024-03-25T19:36:50.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:36:50.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.482+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914135f30 con 0x7fc9141430c0 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823059147 0 0) 0x7fc8f4004eb0 con 0x7fc9140a98f0 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc914135da0 con 0x7fc9140a98f0 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc90c001db0 con 0x7fc9140a98f0 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.485 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.486 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.486 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.486 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.485+0000 7fc910ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:50.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191994943 0 0) 0x7fc914135230 con 0x7fc91413f710 2024-03-25T19:36:50.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8f4004eb0 con 0x7fc91413f710 2024-03-25T19:36:50.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc8fc001870 con 0x7fc91413f710 2024-03-25T19:36:50.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.489+0000 7fc910ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.489 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.499+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868196936 0 0) 0x7fc914135f30 con 0x7fc9141430c0 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.500+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc914135230 con 0x7fc9141430c0 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.500+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc904001b00 con 0x7fc9141430c0 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.500+0000 7fc910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.500+0000 7fc910ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.500+0000 7fc910ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.503 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2518484334 0 0) 0x7fc914135230 con 0x7fc9141430c0 2024-03-25T19:36:50.503 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 >> v1:172.21.15.29:6790/0 conn(0x7fc91413f710 legacy=0x7fc9141dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.503 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 >> v1:172.21.15.29:6789/0 conn(0x7fc9140a98f0 legacy=0x7fc9141d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.503 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 1 monclient: found mon.b 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.503+0000 7fc910ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc910ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9141360c0 con 0x7fc9141430c0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc910ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc910ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc910ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 10 monclient: _renew_subs 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9141d5000 con 0x7fc9141430c0 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 10 monclient: _renew_subs 2024-03-25T19:36:50.504 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9141d5350 con 0x7fc9141430c0 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.504+0000 7fc91a915640 1 librados: init done 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.505+0000 7fc91a915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.505+0000 7fc91a915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.505+0000 7fc91a915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:50.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.505+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc914135090 con 0x7fc9141430c0 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9040011e0 con 0x7fc9141430c0 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc904001c60 con 0x7fc9141430c0 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc910ff9640 10 monclient: dump: 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.815 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.815+0000 7fc9127fc640 4 set_mon_vals no callback set 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.816+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fc90401a900 con 0x7fc9141430c0 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.817+0000 7fc913fff640 10 monclient: get_auth_request con 0x7fc8f406e160 auth_method 0 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.817+0000 7fc910ff9640 10 client.4354.objecter ms_handle_connect 0x7fc8f406e160 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fc904054560 con 0x7fc9141430c0 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 10 client.4354.objecter ms_dispatch 0x7fc914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc90404fea0 con 0x7fc9141430c0 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:50.822+0000 7fc910ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:51.090 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.091+0000 7fc91a915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.091+0000 7fc91a915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.091+0000 7fc91a915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:36:51.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.091+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc91413b700 con 0x7fc9141430c0 2024-03-25T19:36:51.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.131+0000 7fc910ff9640 1 -- 172.21.15.29:0/4170004864 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fc904051fa0 con 0x7fc9141430c0 2024-03-25T19:36:51.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.131+0000 7fc910ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.131+0000 7fc910ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:51.131 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:51.131 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.133+0000 7fc91a915640 10 librados: watch_flush enter 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 10 librados: watch_flush exit 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 10 client.4354.objecter successfully canceled tick 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 >> v1:172.21.15.29:6800/2314944989 conn(0x7fc8f406e160 legacy=0x7fc8f4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 10 monclient: shutdown 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 >> v1:172.21.15.196:6789/0 conn(0x7fc9141430c0 legacy=0x7fc9141db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 shutdown_connections 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 >> 172.21.15.29:0/4170004864 conn(0x7fc9140b7ea0 msgr2=0x7fc91414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 shutdown_connections 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 -- 172.21.15.29:0/4170004864 wait complete. 2024-03-25T19:36:51.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:51.134+0000 7fc91a915640 1 librados: shutdown 2024-03-25T19:36:51.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:57.952 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:58.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.068+0000 7fb65119d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.068+0000 7fb65119d640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.068+0000 7fb65119d640 10 monclient: monmap: 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:36:58.069200+0000 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:36:58.069200+0000 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:58.068 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:36:58.069 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:36:58.069 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.070 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.069+0000 7fb65119d640 1 Processor -- start 2024-03-25T19:36:58.070 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.069+0000 7fb65119d640 1 -- start start 2024-03-25T19:36:58.070 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.069+0000 7fb65119d640 10 monclient: init 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-c con 0x7fb64c0b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-a con 0x7fb64c0b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-b con 0x7fb64c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0af940 con 0x7fb64c0b1730 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0b0a70 con 0x7fb64c0b4d60 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0b0d50 con 0x7fb64c14be90 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.070+0000 7fb65119d640 10 monclient(hunting): authenticate will time out at 6294.823154s 2024-03-25T19:36:58.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.077+0000 7fb64b59e640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fb64c14be90 0x7fb64c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:37198/0 (socket says 172.21.15.29:37198) 2024-03-25T19:36:58.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.077+0000 7fb64b59e640 1 -- 172.21.15.29:0/3792279290 learned_addr learned my addr 172.21.15.29:0/3792279290 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:58.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619641497 0 0) 0x7fb64c0b0a70 con 0x7fb64c0b4d60 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb64c0b4d60 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727716969 0 0) 0x7fb64c0b0d50 con 0x7fb64c14be90 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.099+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c0b0a70 con 0x7fb64c14be90 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517019249 0 0) 0x7fb634003520 con 0x7fb64c0b4d60 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 >> v1:172.21.15.29:6790/0 conn(0x7fb64c14be90 legacy=0x7fb64c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 >> v1:172.21.15.29:6789/0 conn(0x7fb64c0b1730 legacy=0x7fb64c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c0b1030 con 0x7fb64c0b4d60 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb63c001690 con 0x7fb64c0b4d60 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb649d9b640 10 monclient: dump: 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.109 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb65119d640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:36:58.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.108+0000 7fb65119d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:58.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb63c0017f0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 1 -- 172.21.15.29:0/3792279290 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb63c001c00 con 0x7fb64c0b4d60 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb65119d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb65119d640 4 set_mon_vals no callback set 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.128+0000 7fb649d9b640 10 monclient: dump: 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:58.129 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 10 monclient: shutdown 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 1 -- 172.21.15.29:0/3792279290 >> v1:172.21.15.196:6789/0 conn(0x7fb64c0b4d60 legacy=0x7fb64c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 1 -- 172.21.15.29:0/3792279290 shutdown_connections 2024-03-25T19:36:58.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 1 -- 172.21.15.29:0/3792279290 >> 172.21.15.29:0/3792279290 conn(0x7fb64c0617a0 msgr2=0x7fb64c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:58.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 1 -- 172.21.15.29:0/3792279290 shutdown_connections 2024-03-25T19:36:58.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.129+0000 7fb65119d640 1 -- 172.21.15.29:0/3792279290 wait complete. 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.130+0000 7fb65119d640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.130+0000 7fb65119d640 10 monclient: monmap: 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:36:58.132 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.130+0000 7fb65119d640 1 librados: starting msgr at 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.130+0000 7fb65119d640 1 librados: starting objecter 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 Processor -- start 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 -- start start 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 librados: setting wanted keys 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 librados: calling monclient init 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient: init 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-b con 0x7fb64c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-a con 0x7fb64c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient(hunting): picked mon.noname-c con 0x7fb64c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c13b6a0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c137070 con 0x7fb64c0b1730 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c134a00 con 0x7fb64c14be90 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.131+0000 7fb65119d640 10 monclient(hunting): authenticate will time out at 6294.884623s 2024-03-25T19:36:58.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.136+0000 7fb64a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb64c0b4d60 0x7fb64c133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33018/0 (socket says 172.21.15.29:33018) 2024-03-25T19:36:58.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.136+0000 7fb64a59c640 1 -- 172.21.15.29:0/3645597415 learned_addr learned my addr 172.21.15.29:0/3645597415 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:36:58.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.138+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c0b4d60 2024-03-25T19:36:58.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.138+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210311269 0 0) 0x7fb64c13b6a0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:36:58.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.146+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb638004eb0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.156+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c0b1730 2024-03-25T19:36:58.155 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.156+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.164+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174317768 0 0) 0x7fb64c137070 con 0x7fb64c0b1730 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.164+0000 7fb6337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.164+0000 7fb6337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.164+0000 7fb6337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.164+0000 7fb6337fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.165+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c13b6a0 con 0x7fb64c0b1730 2024-03-25T19:36:58.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.167+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c14be90 2024-03-25T19:36:58.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.167+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1608910832 0 0) 0x7fb638004eb0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb64c137070 con 0x7fb64c0b4d60 2024-03-25T19:36:58.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb6400019d0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 10 monclient(hunting): dump: 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.169+0000 7fb6337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2323671721 0 0) 0x7fb64c137070 con 0x7fb64c0b4d60 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 >> v1:172.21.15.29:6790/0 conn(0x7fb64c14be90 legacy=0x7fb64c1342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 >> v1:172.21.15.196:6789/0 conn(0x7fb64c0b1730 legacy=0x7fb64c137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 1 monclient: found mon.a 2024-03-25T19:36:58.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c1372e0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb6337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 5 monclient: authenticate success, global_id 4359 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 10 monclient: _renew_subs 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c1375c0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 10 monclient: _renew_subs 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c1383d0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.181+0000 7fb65119d640 1 librados: init done 2024-03-25T19:36:58.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.182+0000 7fb65119d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.182+0000 7fb65119d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.182+0000 7fb65119d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:58.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.182+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c1379b0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6400035f0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb640003980 con 0x7fb64c0b4d60 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb6337fe640 10 monclient: dump: 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:36:58.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.505+0000 7fb64959a640 4 set_mon_vals no callback set 2024-03-25T19:36:58.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.506+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fb640003ba0 con 0x7fb64c0b4d60 2024-03-25T19:36:58.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.507+0000 7fb64ad9d640 10 monclient: get_auth_request con 0x7fb63806dd10 auth_method 0 2024-03-25T19:36:58.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.507+0000 7fb6337fe640 10 client.4359.objecter ms_handle_connect 0x7fb63806dd10 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fb640054770 con 0x7fb64c0b4d60 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 10 client.4359.objecter ms_dispatch 0x7fb64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb640050280 con 0x7fb64c0b4d60 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.512+0000 7fb6337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:58.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.774+0000 7fb65119d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.774+0000 7fb65119d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.774+0000 7fb65119d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:36:58.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.774+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb64c13b850 con 0x7fb64c0b4d60 2024-03-25T19:36:58.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.827+0000 7fb6337fe640 1 -- 172.21.15.29:0/3645597415 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fb64c13b850 con 0x7fb64c0b4d60 2024-03-25T19:36:58.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.827+0000 7fb6337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.827+0000 7fb6337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:58.827 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:36:58.827 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:58.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 10 librados: watch_flush enter 2024-03-25T19:36:58.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 10 librados: watch_flush exit 2024-03-25T19:36:58.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 10 client.4359.objecter successfully canceled tick 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 >> v1:172.21.15.29:6800/2314944989 conn(0x7fb63806dd10 legacy=0x7fb6380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 10 monclient: shutdown 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 >> v1:172.21.15.29:6789/0 conn(0x7fb64c0b4d60 legacy=0x7fb64c133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 shutdown_connections 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.829+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 >> 172.21.15.29:0/3645597415 conn(0x7fb64c0617a0 msgr2=0x7fb64c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.830+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 shutdown_connections 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.830+0000 7fb65119d640 1 -- 172.21.15.29:0/3645597415 wait complete. 2024-03-25T19:36:58.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:36:58.830+0000 7fb65119d640 1 librados: shutdown 2024-03-25T19:36:58.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:04.791 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:34.954 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.844 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:35.008 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.888 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:35.052 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.898 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:35.061 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.905 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:35.067 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.913 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:35.077 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.924 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:35.088 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.938 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:35.101 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.962 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:35.125 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:05.649 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.763+0000 7f1b299c8640 10 monclient: get_monmap_and_config 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.763+0000 7f1b299c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.763+0000 7f1b299c8640 10 monclient: monmap: 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:05.764540+0000 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:05.764540+0000 2024-03-25T19:37:05.763 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:05.764 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.764 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:05.764 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:05.764 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:05.764 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.764+0000 7f1b299c8640 1 Processor -- start 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.764+0000 7f1b299c8640 1 -- start start 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.764+0000 7f1b299c8640 10 monclient: init 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1b240a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1b2413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:05.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1b241430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:05.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24142520 con 0x7f1b241430c0 2024-03-25T19:37:05.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240b28a0 con 0x7f1b240a98f0 2024-03-25T19:37:05.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240b2a30 con 0x7f1b2413f710 2024-03-25T19:37:05.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:05.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.765+0000 7f1b299c8640 10 monclient(hunting): authenticate will time out at 6302.518224s 2024-03-25T19:37:05.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.776+0000 7f1b237fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1b241430c0 0x7f1b2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37342/0 (socket says 172.21.15.29:37342) 2024-03-25T19:37:05.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.776+0000 7f1b237fe640 1 -- 172.21.15.29:0/460188102 learned_addr learned my addr 172.21.15.29:0/460188102 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:05.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729852064 0 0) 0x7f1b24142520 con 0x7f1b241430c0 2024-03-25T19:37:05.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b10003520 con 0x7f1b241430c0 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163022627 0 0) 0x7f1b240b2a30 con 0x7f1b2413f710 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.794 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:37:05.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.794+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24142520 con 0x7f1b2413f710 2024-03-25T19:37:05.799 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817796390 0 0) 0x7f1b240b28a0 con 0x7f1b240a98f0 2024-03-25T19:37:05.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:37:05.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.800+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b100039d0 con 0x7f1b240a98f0 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134127837 0 0) 0x7f1b24142520 con 0x7f1b2413f710 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 >> v1:172.21.15.196:6789/0 conn(0x7f1b240a98f0 legacy=0x7f1b240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 >> v1:172.21.15.29:6789/0 conn(0x7f1b241430c0 legacy=0x7f1b2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 monclient: found mon.noname-b 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b240b44c0 con 0x7f1b2413f710 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:05.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b0c0015f0 con 0x7f1b2413f710 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b227fc640 10 monclient: dump: 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b299c8640 5 monclient: authenticate success, global_id 4361 2024-03-25T19:37:05.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.805+0000 7f1b299c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b0c001750 con 0x7f1b2413f710 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 1 -- 172.21.15.29:0/460188102 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b0c001ba0 con 0x7f1b2413f710 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b227fc640 10 monclient: dump: 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.822 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:05.823 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:05.823 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:05.823 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b299c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:05.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.822+0000 7f1b299c8640 4 set_mon_vals no callback set 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 10 monclient: shutdown 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 1 -- 172.21.15.29:0/460188102 >> v1:172.21.15.29:6790/0 conn(0x7f1b2413f710 legacy=0x7f1b241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 1 -- 172.21.15.29:0/460188102 shutdown_connections 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 1 -- 172.21.15.29:0/460188102 >> 172.21.15.29:0/460188102 conn(0x7f1b240b7ea0 msgr2=0x7f1b240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 1 -- 172.21.15.29:0/460188102 shutdown_connections 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.823+0000 7f1b299c8640 1 -- 172.21.15.29:0/460188102 wait complete. 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.824+0000 7f1b299c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:05.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.824+0000 7f1b299c8640 10 monclient: monmap: 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.824+0000 7f1b299c8640 1 librados: starting msgr at 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.824+0000 7f1b299c8640 1 librados: starting objecter 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.824+0000 7f1b299c8640 1 Processor -- start 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 -- start start 2024-03-25T19:37:05.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 librados: setting wanted keys 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 librados: calling monclient init 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient: init 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1b240a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1b241430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240abe30 con 0x7f1b241430c0 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b241d7590 con 0x7f1b240a98f0 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b241d7870 con 0x7f1b2413f710 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:05.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.825+0000 7f1b299c8640 10 monclient(hunting): authenticate will time out at 6302.578658s 2024-03-25T19:37:05.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.830+0000 7f1b22ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1b240a98f0 0x7f1b241d7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:58180/0 (socket says 172.21.15.29:58180) 2024-03-25T19:37:05.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.830+0000 7f1b22ffd640 1 -- 172.21.15.29:0/1177677632 learned_addr learned my addr 172.21.15.29:0/1177677632 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:05.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.830+0000 7f1b20ff9640 10 client.?.objecter ms_handle_connect 0x7f1b240a98f0 2024-03-25T19:37:05.830 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.830+0000 7f1b20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.839+0000 7f1b20ff9640 10 client.?.objecter ms_handle_connect 0x7f1b2413f710 2024-03-25T19:37:05.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.839+0000 7f1b20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.852 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.853+0000 7f1b20ff9640 10 client.?.objecter ms_handle_connect 0x7f1b241430c0 2024-03-25T19:37:05.852 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.853+0000 7f1b20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955810593 0 0) 0x7f1b241d7870 con 0x7f1b2413f710 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:37:05.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.926+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04004eb0 con 0x7f1b2413f710 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122090477 0 0) 0x7f1b240abe30 con 0x7f1b241430c0 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:37:05.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.929+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b241d7870 con 0x7f1b241430c0 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620562008 0 0) 0x7f1b241d7590 con 0x7f1b240a98f0 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:37:05.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.931+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b240abe30 con 0x7f1b240a98f0 2024-03-25T19:37:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325996879 0 0) 0x7f1b04004eb0 con 0x7f1b2413f710 2024-03-25T19:37:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b241d7590 con 0x7f1b2413f710 2024-03-25T19:37:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b0c001a60 con 0x7f1b2413f710 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.948+0000 7f1b20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.953+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616448401 0 0) 0x7f1b240abe30 con 0x7f1b240a98f0 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.953+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b04004eb0 con 0x7f1b240a98f0 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.954+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b140019b0 con 0x7f1b240a98f0 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.954+0000 7f1b20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.954+0000 7f1b20ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.954+0000 7f1b20ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.953 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:05.954 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067728912 0 0) 0x7f1b241d7870 con 0x7f1b241430c0 2024-03-25T19:37:05.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b240abe30 con 0x7f1b241430c0 2024-03-25T19:37:05.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b1c001a70 con 0x7f1b241430c0 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.959+0000 7f1b20ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:05.959 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 129131590 0 0) 0x7f1b04004eb0 con 0x7f1b240a98f0 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 >> v1:172.21.15.29:6790/0 conn(0x7f1b2413f710 legacy=0x7f1b241d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 >> v1:172.21.15.29:6789/0 conn(0x7f1b241430c0 legacy=0x7f1b24138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 1 monclient: found mon.b 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b241d8410 con 0x7f1b240a98f0 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 10 monclient: _renew_subs 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b241d50a0 con 0x7f1b240a98f0 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 10 monclient: _renew_subs 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b24138200 con 0x7f1b240a98f0 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.972+0000 7f1b299c8640 1 librados: init done 2024-03-25T19:37:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.973+0000 7f1b299c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.973+0000 7f1b299c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.973+0000 7f1b299c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:05.973+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b240627b0 con 0x7f1b240a98f0 2024-03-25T19:37:06.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b14003610 con 0x7f1b240a98f0 2024-03-25T19:37:06.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1b14003960 con 0x7f1b240a98f0 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b20ff9640 10 monclient: dump: 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:06.258 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:06.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.258+0000 7f1b21ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:06.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.259+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f1b14003b80 con 0x7f1b240a98f0 2024-03-25T19:37:06.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.259+0000 7f1b1a7fc640 10 monclient: get_auth_request con 0x7f1b0406e110 auth_method 0 2024-03-25T19:37:06.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.259+0000 7f1b20ff9640 10 client.4366.objecter ms_handle_connect 0x7f1b0406e110 2024-03-25T19:37:06.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.263+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f1b14054710 con 0x7f1b240a98f0 2024-03-25T19:37:06.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.263+0000 7f1b20ff9640 10 client.4366.objecter ms_dispatch 0x7f1b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:06.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.263+0000 7f1b20ff9640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:06.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.263+0000 7f1b20ff9640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:06.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.263+0000 7f1b20ff9640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:37:06.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.268+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b14050220 con 0x7f1b240a98f0 2024-03-25T19:37:06.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.268+0000 7f1b20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.268+0000 7f1b20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:06.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.516+0000 7f1b299c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.516+0000 7f1b299c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.516+0000 7f1b299c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:06.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.516+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b240b71a0 con 0x7f1b240a98f0 2024-03-25T19:37:06.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.566+0000 7f1b20ff9640 1 -- 172.21.15.29:0/1177677632 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f1b14052150 con 0x7f1b240a98f0 2024-03-25T19:37:06.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.566+0000 7f1b20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.566+0000 7f1b20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:06.566 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:06.566 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 10 librados: watch_flush enter 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 10 librados: watch_flush exit 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 >> v1:172.21.15.29:6800/2314944989 conn(0x7f1b0406e110 legacy=0x7f1b040705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 10 monclient: shutdown 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 >> v1:172.21.15.196:6789/0 conn(0x7f1b240a98f0 legacy=0x7f1b241d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 shutdown_connections 2024-03-25T19:37:06.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 >> 172.21.15.29:0/1177677632 conn(0x7f1b240b7ea0 msgr2=0x7f1b24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.568+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 shutdown_connections 2024-03-25T19:37:06.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.569+0000 7f1b299c8640 1 -- 172.21.15.29:0/1177677632 wait complete. 2024-03-25T19:37:06.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:06.569+0000 7f1b299c8640 1 librados: shutdown 2024-03-25T19:37:06.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:13.387 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:13.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.490+0000 7fdbb4a1e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.490+0000 7fdbb4a1e640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.490+0000 7fdbb4a1e640 10 monclient: monmap: 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:13.492347+0000 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:13.492347+0000 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:13.491 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 1 Processor -- start 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 1 -- start start 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient: init 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb00b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb00b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb014be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00af940 con 0x7fdbb00b1730 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00b0a70 con 0x7fdbb014be90 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00b0d50 con 0x7fdbb00b4d60 2024-03-25T19:37:13.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.491+0000 7fdbb4a1e640 10 monclient(hunting): authenticate will time out at 6310.245766s 2024-03-25T19:37:13.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.492+0000 7fdbaed9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdbb014be90 0x7fdbb014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50312/0 (socket says 172.21.15.29:50312) 2024-03-25T19:37:13.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.492+0000 7fdbaed9d640 1 -- 172.21.15.29:0/2348316309 learned_addr learned my addr 172.21.15.29:0/2348316309 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:13.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417981069 0 0) 0x7fdbb00af940 con 0x7fdbb00b1730 2024-03-25T19:37:13.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:37:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.505+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c003520 con 0x7fdbb00b1730 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841528870 0 0) 0x7fdb9c003520 con 0x7fdbb00b1730 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 >> v1:172.21.15.29:6790/0 conn(0x7fdbb00b4d60 legacy=0x7fdbb013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 >> v1:172.21.15.196:6789/0 conn(0x7fdbb014be90 legacy=0x7fdbb014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 monclient: found mon.noname-a 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00b1030 con 0x7fdbb00b1730 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb98001690 con 0x7fdbb00b1730 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbb4a1e640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbb4a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbadd9b640 10 monclient: dump: 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.535 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.536 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.533+0000 7fdbb4a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb980017f0 con 0x7fdbb00b1730 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 1 -- 172.21.15.29:0/2348316309 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb98001c00 con 0x7fdbb00b1730 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbadd9b640 10 monclient: dump: 2024-03-25T19:37:13.546 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbb4a1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.545+0000 7fdbb4a1e640 4 set_mon_vals no callback set 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 10 monclient: shutdown 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/2348316309 >> v1:172.21.15.29:6789/0 conn(0x7fdbb00b1730 legacy=0x7fdbb00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/2348316309 shutdown_connections 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/2348316309 >> 172.21.15.29:0/2348316309 conn(0x7fdbb00617a0 msgr2=0x7fdbb0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/2348316309 shutdown_connections 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.546+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/2348316309 wait complete. 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.547+0000 7fdbb4a1e640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.547+0000 7fdbb4a1e640 10 monclient: monmap: 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:13.549 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 librados: starting msgr at 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 librados: starting objecter 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 Processor -- start 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 -- start start 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 librados: setting wanted keys 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 1 librados: calling monclient init 2024-03-25T19:37:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.548+0000 7fdbb4a1e640 10 monclient: init 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb00b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb00b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb0140380 con 0x7fdbb00b4d60 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb0141ca0 con 0x7fdbb00b1730 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb0140860 con 0x7fdbb014be90 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.549+0000 7fdbb4a1e640 10 monclient(hunting): authenticate will time out at 6310.303316s 2024-03-25T19:37:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.555+0000 7fdba7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdbb00b4d60 0x7fdbb01db720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48902/0 (socket says 172.21.15.29:48902) 2024-03-25T19:37:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.555+0000 7fdba7fff640 1 -- 172.21.15.29:0/1970467053 learned_addr learned my addr 172.21.15.29:0/1970467053 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.556+0000 7fdba6ffd640 10 client.?.objecter ms_handle_connect 0x7fdbb00b4d60 2024-03-25T19:37:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.556+0000 7fdba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906363164 0 0) 0x7fdbb0140380 con 0x7fdbb00b4d60 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.570+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90004eb0 con 0x7fdbb00b4d60 2024-03-25T19:37:13.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.571+0000 7fdba6ffd640 10 client.?.objecter ms_handle_connect 0x7fdbb00b1730 2024-03-25T19:37:13.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.571+0000 7fdba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.578+0000 7fdba6ffd640 10 client.?.objecter ms_handle_connect 0x7fdbb014be90 2024-03-25T19:37:13.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.578+0000 7fdba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 569439861 0 0) 0x7fdb90004eb0 con 0x7fdbb00b4d60 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb0140380 con 0x7fdbb00b4d60 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb98001690 con 0x7fdbb00b4d60 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:13.582 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:13.583 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:13.583 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:13.583 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.581+0000 7fdba6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606773499 0 0) 0x7fdbb0141ca0 con 0x7fdbb00b1730 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:37:13.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.588+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90004eb0 con 0x7fdbb00b1730 2024-03-25T19:37:13.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1201052006 0 0) 0x7fdbb0140380 con 0x7fdbb00b4d60 2024-03-25T19:37:13.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 >> v1:172.21.15.29:6790/0 conn(0x7fdbb014be90 legacy=0x7fdbb01e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 >> v1:172.21.15.196:6789/0 conn(0x7fdbb00b1730 legacy=0x7fdbb01db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 1 monclient: found mon.a 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb01423a0 con 0x7fdbb00b4d60 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 10 monclient: _renew_subs 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0138650 con 0x7fdbb00b4d60 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 10 monclient: _renew_subs 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0140d00 con 0x7fdbb00b4d60 2024-03-25T19:37:13.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.593+0000 7fdbb4a1e640 1 librados: init done 2024-03-25T19:37:13.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.594+0000 7fdbb4a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.594+0000 7fdbb4a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.594+0000 7fdbb4a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:37:13.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.594+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb00627b0 con 0x7fdbb00b4d60 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb98003a20 con 0x7fdbb00b4d60 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fdb98003e70 con 0x7fdbb00b4d60 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdba6ffd640 10 monclient: dump: 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.915+0000 7fdbad59a640 4 set_mon_vals no callback set 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.916+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fdb98004090 con 0x7fdbb00b4d60 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.916+0000 7fdbae59c640 10 monclient: get_auth_request con 0x7fdb9006ddd0 auth_method 0 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.916+0000 7fdba6ffd640 10 client.4371.objecter ms_handle_connect 0x7fdb9006ddd0 2024-03-25T19:37:13.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fdb98054a90 con 0x7fdbb00b4d60 2024-03-25T19:37:13.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 10 client.4371.objecter ms_dispatch 0x7fdbb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:13.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:37:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb980505a0 con 0x7fdbb00b4d60 2024-03-25T19:37:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:13.921+0000 7fdba6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:14.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.178+0000 7fdbb4a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.178+0000 7fdbb4a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.178+0000 7fdbb4a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:37:14.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.178+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbb0069f40 con 0x7fdbb00b4d60 2024-03-25T19:37:14.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.200+0000 7fdba6ffd640 1 -- 172.21.15.29:0/1970467053 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fdb980524d0 con 0x7fdbb00b4d60 2024-03-25T19:37:14.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.200+0000 7fdba6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.200 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.200+0000 7fdba6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:14.201 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:14.201 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.202+0000 7fdbb4a1e640 10 librados: watch_flush enter 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 10 librados: watch_flush exit 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 >> v1:172.21.15.29:6800/2314944989 conn(0x7fdb9006ddd0 legacy=0x7fdb90070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 10 monclient: shutdown 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 >> v1:172.21.15.29:6789/0 conn(0x7fdbb00b4d60 legacy=0x7fdbb01db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 shutdown_connections 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 >> 172.21.15.29:0/1970467053 conn(0x7fdbb00617a0 msgr2=0x7fdbb013c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 shutdown_connections 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 -- 172.21.15.29:0/1970467053 wait complete. 2024-03-25T19:37:14.205 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:14.203+0000 7fdbb4a1e640 1 librados: shutdown 2024-03-25T19:37:14.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:21.022 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:21.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.140+0000 7fae830f6640 10 monclient: get_monmap_and_config 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.140+0000 7fae830f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.140+0000 7fae830f6640 10 monclient: monmap: 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:21.142143+0000 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:21.142143+0000 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:21.141 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.141+0000 7fae830f6640 1 Processor -- start 2024-03-25T19:37:21.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.141+0000 7fae830f6640 1 -- start start 2024-03-25T19:37:21.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.141+0000 7fae830f6640 10 monclient: init 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.141+0000 7fae830f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.141+0000 7fae830f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c142520 con 0x7fae7c13f710 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0b28a0 con 0x7fae7c1430c0 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0b2a30 con 0x7fae7c0a98f0 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.142+0000 7fae830f6640 10 monclient(hunting): authenticate will time out at 6317.895990s 2024-03-25T19:37:21.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.145+0000 7fae81693640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fae7c1430c0 0x7fae7c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50842/0 (socket says 172.21.15.29:50842) 2024-03-25T19:37:21.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.145+0000 7fae81693640 1 -- 172.21.15.29:0/660524489 learned_addr learned my addr 172.21.15.29:0/660524489 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:21.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687630450 0 0) 0x7fae7c0b28a0 con 0x7fae7c1430c0 2024-03-25T19:37:21.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:37:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.156+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae6c003520 con 0x7fae7c1430c0 2024-03-25T19:37:21.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550305790 0 0) 0x7fae7c142520 con 0x7fae7c13f710 2024-03-25T19:37:21.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:37:21.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.165+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c0b28a0 con 0x7fae7c13f710 2024-03-25T19:37:21.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253117409 0 0) 0x7fae6c003520 con 0x7fae7c1430c0 2024-03-25T19:37:21.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 >> v1:172.21.15.29:6790/0 conn(0x7fae7c0a98f0 legacy=0x7fae7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 >> v1:172.21.15.29:6789/0 conn(0x7fae7c13f710 legacy=0x7fae7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0b44c0 con 0x7fae7c1430c0 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fae74001a10 con 0x7fae7c1430c0 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae830f6640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae830f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae7b7fe640 10 monclient: dump: 2024-03-25T19:37:21.176 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.175+0000 7fae830f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae74001b70 con 0x7fae7c1430c0 2024-03-25T19:37:21.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 1 -- 172.21.15.29:0/660524489 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fae74001f80 con 0x7fae7c1430c0 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae830f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae830f6640 4 set_mon_vals no callback set 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.188+0000 7fae7b7fe640 10 monclient: dump: 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:21.189 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 10 monclient: shutdown 2024-03-25T19:37:21.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:21.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 1 -- 172.21.15.29:0/660524489 >> v1:172.21.15.196:6789/0 conn(0x7fae7c1430c0 legacy=0x7fae7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 1 -- 172.21.15.29:0/660524489 shutdown_connections 2024-03-25T19:37:21.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 1 -- 172.21.15.29:0/660524489 >> 172.21.15.29:0/660524489 conn(0x7fae7c0b7ea0 msgr2=0x7fae7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 1 -- 172.21.15.29:0/660524489 shutdown_connections 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.189+0000 7fae830f6640 1 -- 172.21.15.29:0/660524489 wait complete. 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.190+0000 7fae830f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.190+0000 7fae830f6640 10 monclient: monmap: 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:21.191 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.190+0000 7fae830f6640 1 librados: starting msgr at 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.190+0000 7fae830f6640 1 librados: starting objecter 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.190+0000 7fae830f6640 1 Processor -- start 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.191+0000 7fae830f6640 1 -- start start 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.191+0000 7fae830f6640 1 librados: setting wanted keys 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.191+0000 7fae830f6640 1 librados: calling monclient init 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.191+0000 7fae830f6640 10 monclient: init 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:21.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c1db170 con 0x7fae7c13f710 2024-03-25T19:37:21.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c1dac50 con 0x7fae7c0a98f0 2024-03-25T19:37:21.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c1dade0 con 0x7fae7c1430c0 2024-03-25T19:37:21.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.192+0000 7fae830f6640 10 monclient(hunting): authenticate will time out at 6317.946234s 2024-03-25T19:37:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.201+0000 7fae80e92640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fae7c0a98f0 0x7fae7c1394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50884/0 (socket says 172.21.15.29:50884) 2024-03-25T19:37:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.201+0000 7fae80e92640 1 -- 172.21.15.29:0/1610784905 learned_addr learned my addr 172.21.15.29:0/1610784905 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:21.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.204+0000 7fae797fa640 10 client.?.objecter ms_handle_connect 0x7fae7c0a98f0 2024-03-25T19:37:21.204 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.204+0000 7fae797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.206+0000 7fae797fa640 10 client.?.objecter ms_handle_connect 0x7fae7c1430c0 2024-03-25T19:37:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.206+0000 7fae797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.217 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.217+0000 7fae797fa640 10 client.?.objecter ms_handle_connect 0x7fae7c13f710 2024-03-25T19:37:21.218 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.217+0000 7fae797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733697325 0 0) 0x7fae7c1dade0 con 0x7fae7c1430c0 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:37:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.220+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae60004eb0 con 0x7fae7c1430c0 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981149906 0 0) 0x7fae7c1db170 con 0x7fae7c13f710 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c1dade0 con 0x7fae7c13f710 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260807859 0 0) 0x7fae7c1dac50 con 0x7fae7c0a98f0 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:37:21.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.227+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c1db170 con 0x7fae7c0a98f0 2024-03-25T19:37:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534415454 0 0) 0x7fae60004eb0 con 0x7fae7c1430c0 2024-03-25T19:37:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae7c1dac50 con 0x7fae7c1430c0 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fae74001e80 con 0x7fae7c1430c0 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 10 monclient(hunting): dump: 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.233 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.232+0000 7fae797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.242+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1632305542 0 0) 0x7fae7c1db170 con 0x7fae7c0a98f0 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.242+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae60004eb0 con 0x7fae7c0a98f0 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.242+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fae70001970 con 0x7fae7c0a98f0 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.242+0000 7fae797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.243+0000 7fae797fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.243+0000 7fae797fa640 10 monclient(hunting): dump: 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.243 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:21.244 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2822116032 0 0) 0x7fae7c1dac50 con 0x7fae7c1430c0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 >> v1:172.21.15.196:6789/0 conn(0x7fae7c0a98f0 legacy=0x7fae7c1394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 >> v1:172.21.15.29:6789/0 conn(0x7fae7c13f710 legacy=0x7fae7c1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 1 monclient: found mon.c 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c13c170 con 0x7fae7c1430c0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 10 monclient: _renew_subs 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:21.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c13c420 con 0x7fae7c1430c0 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 10 monclient: _renew_subs 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c133860 con 0x7fae7c1430c0 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.245+0000 7fae830f6640 1 librados: init done 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.246+0000 7fae830f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.246+0000 7fae830f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.246+0000 7fae830f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:21.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.246+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c1daf70 con 0x7fae7c1430c0 2024-03-25T19:37:21.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.461+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae74002020 con 0x7fae7c1430c0 2024-03-25T19:37:21.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.461+0000 7fae797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.461+0000 7fae7affd640 4 set_mon_vals no callback set 2024-03-25T19:37:21.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.552+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fae74004fb0 con 0x7fae7c1430c0 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.552+0000 7fae797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.552+0000 7fae797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.552+0000 7fae797fa640 10 monclient: dump: 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:21.553 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:21.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.553+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fae7401af80 con 0x7fae7c1430c0 2024-03-25T19:37:21.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.554+0000 7fae80e92640 10 monclient: get_auth_request con 0x7fae6006df20 auth_method 0 2024-03-25T19:37:21.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.554+0000 7fae797fa640 10 client.4376.objecter ms_handle_connect 0x7fae6006df20 2024-03-25T19:37:21.561 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fae74054af0 con 0x7fae7c1430c0 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 10 client.4376.objecter ms_dispatch 0x7fae7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae74050600 con 0x7fae7c1430c0 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.561+0000 7fae797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:21.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.817+0000 7fae830f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.817+0000 7fae830f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.817+0000 7fae830f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:21.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.817+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae7c0627b0 con 0x7fae7c1430c0 2024-03-25T19:37:21.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.854+0000 7fae797fa640 1 -- 172.21.15.29:0/1610784905 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fae74052530 con 0x7fae7c1430c0 2024-03-25T19:37:21.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.854+0000 7fae797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.854+0000 7fae797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:21.855 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:21.855 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:21.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 10 librados: watch_flush enter 2024-03-25T19:37:21.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 10 librados: watch_flush exit 2024-03-25T19:37:21.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T19:37:21.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 10 client.4376.objecter successfully canceled tick 2024-03-25T19:37:21.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 >> v1:172.21.15.29:6800/2314944989 conn(0x7fae6006df20 legacy=0x7fae600703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 10 monclient: shutdown 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.857+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 >> v1:172.21.15.29:6790/0 conn(0x7fae7c1430c0 legacy=0x7fae7c132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.858+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 shutdown_connections 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.858+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 >> 172.21.15.29:0/1610784905 conn(0x7fae7c0b7ea0 msgr2=0x7fae7c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.858+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 shutdown_connections 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.858+0000 7fae830f6640 1 -- 172.21.15.29:0/1610784905 wait complete. 2024-03-25T19:37:21.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:21.858+0000 7fae830f6640 1 librados: shutdown 2024-03-25T19:37:21.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:24.880 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:55.043 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.882 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:55.046 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.959 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:55.123 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.984 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:55.146 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:25.000 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:55.162 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:25.017 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:55.181 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:25.021 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:55.185 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:25.029 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:03:55.192 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:25.056 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:55.220 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:28.676 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.793+0000 7f8e1693a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.793+0000 7f8e1693a640 10 monclient: build_initial_monmap 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.794+0000 7f8e1693a640 10 monclient: monmap: 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:28.793 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:28.794792+0000 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:28.794792+0000 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:28.794 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.794+0000 7f8e1693a640 1 Processor -- start 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.794+0000 7f8e1693a640 1 -- start start 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.794+0000 7f8e1693a640 10 monclient: init 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e100b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100af940 con 0x7f8e100b4d60 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100b0a70 con 0x7f8e100b1730 2024-03-25T19:37:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100b0d50 con 0x7f8e1014be90 2024-03-25T19:37:28.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:28.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.795+0000 7f8e1693a640 10 monclient(hunting): authenticate will time out at 6325.548554s 2024-03-25T19:37:28.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.806+0000 7f8e0f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8e100b1730 0x7f8e100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41586/0 (socket says 172.21.15.29:41586) 2024-03-25T19:37:28.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.806+0000 7f8e0f7fe640 1 -- 172.21.15.29:0/1781189189 learned_addr learned my addr 172.21.15.29:0/1781189189 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949329439 0 0) 0x7f8e100b0a70 con 0x7f8e100b1730 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:37:28.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.820+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df8003520 con 0x7f8e100b1730 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428960088 0 0) 0x7f8e100af940 con 0x7f8e100b4d60 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:37:28.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.831+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e100b0a70 con 0x7f8e100b4d60 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352928455 0 0) 0x7f8e100b0d50 con 0x7f8e1014be90 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:37:28.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.834+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e100af940 con 0x7f8e1014be90 2024-03-25T19:37:28.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1820833174 0 0) 0x7f8df8003520 con 0x7f8e100b1730 2024-03-25T19:37:28.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 >> v1:172.21.15.29:6790/0 conn(0x7f8e1014be90 legacy=0x7f8e1014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 >> v1:172.21.15.29:6789/0 conn(0x7f8e100b4d60 legacy=0x7f8e1013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 monclient: found mon.noname-c 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100b1030 con 0x7f8e100b1730 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: _finish_auth 0 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8e04001690 con 0x7f8e100b1730 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e0effd640 10 monclient: dump: 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:28.836 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e1693a640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:37:28.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.835+0000 7f8e1693a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e040017f0 con 0x7f8e100b1730 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 1 -- 172.21.15.29:0/1781189189 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8e04001c00 con 0x7f8e100b1730 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e0effd640 10 monclient: dump: 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:28.846 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e1693a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:28.847 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e1693a640 4 set_mon_vals no callback set 2024-03-25T19:37:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e1693a640 10 monclient: shutdown 2024-03-25T19:37:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.847+0000 7f8e1693a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 1 -- 172.21.15.29:0/1781189189 >> v1:172.21.15.196:6789/0 conn(0x7f8e100b1730 legacy=0x7f8e100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 1 -- 172.21.15.29:0/1781189189 shutdown_connections 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 1 -- 172.21.15.29:0/1781189189 >> 172.21.15.29:0/1781189189 conn(0x7f8e100617a0 msgr2=0x7f8e10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 1 -- 172.21.15.29:0/1781189189 shutdown_connections 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 1 -- 172.21.15.29:0/1781189189 wait complete. 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 10 monclient: build_initial_monmap 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.848+0000 7f8e1693a640 10 monclient: monmap: 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:28.849 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 librados: starting msgr at 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 librados: starting objecter 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 Processor -- start 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 -- start start 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 librados: setting wanted keys 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 1 librados: calling monclient init 2024-03-25T19:37:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.849+0000 7f8e1693a640 10 monclient: init 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e100b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e100b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e101380a0 con 0x7f8e100b1730 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1012f7b0 con 0x7f8e100b4d60 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1013c730 con 0x7f8e1014be90 2024-03-25T19:37:28.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:28.852 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.850+0000 7f8e1693a640 10 monclient(hunting): authenticate will time out at 6325.603264s 2024-03-25T19:37:28.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.867+0000 7f8e0f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8e100b4d60 0x7f8e1012e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41652/0 (socket says 172.21.15.29:41652) 2024-03-25T19:37:28.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.867+0000 7f8e0f7fe640 1 -- 172.21.15.29:0/438687497 learned_addr learned my addr 172.21.15.29:0/438687497 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:28.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.872+0000 7f8e0cff9640 10 client.?.objecter ms_handle_connect 0x7f8e100b1730 2024-03-25T19:37:28.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.872+0000 7f8e0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.872+0000 7f8e0cff9640 10 client.?.objecter ms_handle_connect 0x7f8e1014be90 2024-03-25T19:37:28.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.872+0000 7f8e0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.877+0000 7f8e0cff9640 10 client.?.objecter ms_handle_connect 0x7f8e100b4d60 2024-03-25T19:37:28.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.877+0000 7f8e0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745357081 0 0) 0x7f8e1013c730 con 0x7f8e1014be90 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:37:28.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.879+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dfc004eb0 con 0x7f8e1014be90 2024-03-25T19:37:28.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160862335 0 0) 0x7f8e101380a0 con 0x7f8e100b1730 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.884+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1013c730 con 0x7f8e100b1730 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309350862 0 0) 0x7f8e1012f7b0 con 0x7f8e100b4d60 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:37:28.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.885+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e101380a0 con 0x7f8e100b4d60 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063329104 0 0) 0x7f8dfc004eb0 con 0x7f8e1014be90 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e1012f7b0 con 0x7f8e1014be90 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8e08001bd0 con 0x7f8e1014be90 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 10 monclient(hunting): dump: 2024-03-25T19:37:28.890 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:28.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.890+0000 7f8e0cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:28.899 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 595119501 0 0) 0x7f8e1012f7b0 con 0x7f8e1014be90 2024-03-25T19:37:28.899 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 >> v1:172.21.15.196:6789/0 conn(0x7f8e100b4d60 legacy=0x7f8e1012e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 >> v1:172.21.15.29:6789/0 conn(0x7f8e100b1730 legacy=0x7f8e10135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 1 monclient: found mon.c 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10139120 con 0x7f8e1014be90 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 5 monclient: authenticate success, global_id 4382 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 10 monclient: _renew_subs 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e101457a0 con 0x7f8e1014be90 2024-03-25T19:37:28.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 10 monclient: _renew_subs 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10136620 con 0x7f8e1014be90 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.900+0000 7f8e1693a640 1 librados: init done 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.901+0000 7f8e1693a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.901+0000 7f8e1693a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.901+0000 7f8e1693a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:28.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.901+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e101392b0 con 0x7f8e1014be90 2024-03-25T19:37:28.910 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.910+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e08002060 con 0x7f8e1014be90 2024-03-25T19:37:28.910 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.910+0000 7f8e0cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:28.910 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:28.910+0000 7f8e0e7fc640 4 set_mon_vals no callback set 2024-03-25T19:37:29.223 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.223+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8e08003a50 con 0x7f8e1014be90 2024-03-25T19:37:29.223 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.223+0000 7f8e0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.223 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.223+0000 7f8e0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:29.223 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.223+0000 7f8e0cff9640 10 monclient: dump: 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:29.224 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:29.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.224+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f8e08003c30 con 0x7f8e1014be90 2024-03-25T19:37:29.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.225+0000 7f8e0ffff640 10 monclient: get_auth_request con 0x7f8dfc06ddf0 auth_method 0 2024-03-25T19:37:29.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.225+0000 7f8e0cff9640 10 client.4382.objecter ms_handle_connect 0x7f8dfc06ddf0 2024-03-25T19:37:29.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f8e080536c0 con 0x7f8e1014be90 2024-03-25T19:37:29.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 10 client.4382.objecter ms_dispatch 0x7f8e10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:29.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:29.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:29.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T19:37:29.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e0804f000 con 0x7f8e1014be90 2024-03-25T19:37:29.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.231+0000 7f8e0cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:29.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.463+0000 7f8e1693a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.463+0000 7f8e1693a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.463+0000 7f8e1693a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:29.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.463+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e100afbd0 con 0x7f8e1014be90 2024-03-25T19:37:29.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.492+0000 7f8e0cff9640 1 -- 172.21.15.29:0/438687497 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f8e08051100 con 0x7f8e1014be90 2024-03-25T19:37:29.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.492+0000 7f8e0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.492+0000 7f8e0cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:29.491 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:29.492 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:29.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 10 librados: watch_flush enter 2024-03-25T19:37:29.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 10 librados: watch_flush exit 2024-03-25T19:37:29.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T19:37:29.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 10 client.4382.objecter successfully canceled tick 2024-03-25T19:37:29.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 >> v1:172.21.15.29:6800/2314944989 conn(0x7f8dfc06ddf0 legacy=0x7f8dfc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 10 monclient: shutdown 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 >> v1:172.21.15.29:6790/0 conn(0x7f8e1014be90 legacy=0x7f8e1012f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 shutdown_connections 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 >> 172.21.15.29:0/438687497 conn(0x7f8e100617a0 msgr2=0x7f8e100b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 shutdown_connections 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 -- 172.21.15.29:0/438687497 wait complete. 2024-03-25T19:37:29.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:29.494+0000 7f8e1693a640 1 librados: shutdown 2024-03-25T19:37:29.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:36.312 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:36.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.431+0000 7f38a6825640 10 monclient: get_monmap_and_config 2024-03-25T19:37:36.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.431+0000 7f38a6825640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.431+0000 7f38a6825640 10 monclient: monmap: 2024-03-25T19:37:36.430 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:36.430 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:36.431934+0000 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:36.431934+0000 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.432+0000 7f38a6825640 1 Processor -- start 2024-03-25T19:37:36.431 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.432+0000 7f38a6825640 1 -- start start 2024-03-25T19:37:36.432 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.432+0000 7f38a6825640 10 monclient: init 2024-03-25T19:37:36.432 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-b con 0x7f38a00a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-c con 0x7f38a013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-a con 0x7f38a01430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a0142520 con 0x7f38a01430c0 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a00b28a0 con 0x7f38a013f710 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a00b2a30 con 0x7f38a00a98f0 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.433+0000 7f38a6825640 10 monclient(hunting): authenticate will time out at 6333.186240s 2024-03-25T19:37:36.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.440+0000 7f389f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f38a013f710 0x7f38a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36326/0 (socket says 172.21.15.29:36326) 2024-03-25T19:37:36.439 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.440+0000 7f389f7fe640 1 -- 172.21.15.29:0/442143468 learned_addr learned my addr 172.21.15.29:0/442143468 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209840559 0 0) 0x7f38a00b2a30 con 0x7f38a00a98f0 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:37:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.454+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888003520 con 0x7f38a00a98f0 2024-03-25T19:37:36.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735286383 0 0) 0x7f38a00b28a0 con 0x7f38a013f710 2024-03-25T19:37:36.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:37:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.465+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a00b2a30 con 0x7f38a013f710 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960940664 0 0) 0x7f3888003520 con 0x7f38a00a98f0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 >> v1:172.21.15.196:6789/0 conn(0x7f38a013f710 legacy=0x7f38a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 >> v1:172.21.15.29:6789/0 conn(0x7f38a01430c0 legacy=0x7f38a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 monclient: found mon.noname-b 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a00b44c0 con 0x7f38a00a98f0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3890001690 con 0x7f38a00a98f0 2024-03-25T19:37:36.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f389effd640 10 monclient: dump: 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f38a6825640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:37:36.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.466+0000 7f38a6825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38900017f0 con 0x7f38a00a98f0 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 1 -- 172.21.15.29:0/442143468 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3890001c00 con 0x7f38a00a98f0 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f38a6825640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f38a6825640 4 set_mon_vals no callback set 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f389effd640 10 monclient: dump: 2024-03-25T19:37:36.482 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:36.483 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f38a6825640 10 monclient: shutdown 2024-03-25T19:37:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f38a6825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.483+0000 7f38a6825640 1 -- 172.21.15.29:0/442143468 >> v1:172.21.15.29:6790/0 conn(0x7f38a00a98f0 legacy=0x7f38a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.484+0000 7f38a6825640 1 -- 172.21.15.29:0/442143468 shutdown_connections 2024-03-25T19:37:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.484+0000 7f38a6825640 1 -- 172.21.15.29:0/442143468 >> 172.21.15.29:0/442143468 conn(0x7f38a00b7ea0 msgr2=0x7f38a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.484+0000 7f38a6825640 1 -- 172.21.15.29:0/442143468 shutdown_connections 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.484+0000 7f38a6825640 1 -- 172.21.15.29:0/442143468 wait complete. 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 10 monclient: monmap: 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 librados: starting msgr at 2024-03-25T19:37:36.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 librados: starting objecter 2024-03-25T19:37:36.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 Processor -- start 2024-03-25T19:37:36.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 -- start start 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 librados: setting wanted keys 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 1 librados: calling monclient init 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.485+0000 7f38a6825640 10 monclient: init 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-b con 0x7f38a00a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-a con 0x7f38a013f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient(hunting): picked mon.noname-c con 0x7f38a01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a013be70 con 0x7f38a013f710 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a0131660 con 0x7f38a00a98f0 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a01317f0 con 0x7f38a01430c0 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.486+0000 7f38a6825640 10 monclient(hunting): authenticate will time out at 6333.239382s 2024-03-25T19:37:36.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.495+0000 7f389ffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f38a00a98f0 0x7f38a0130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36370/0 (socket says 172.21.15.29:36370) 2024-03-25T19:37:36.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.495+0000 7f389ffff640 1 -- 172.21.15.29:0/3121214169 learned_addr learned my addr 172.21.15.29:0/3121214169 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:36.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.498+0000 7f389cff9640 10 client.?.objecter ms_handle_connect 0x7f38a00a98f0 2024-03-25T19:37:36.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.498+0000 7f389cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.506+0000 7f389cff9640 10 client.?.objecter ms_handle_connect 0x7f38a01430c0 2024-03-25T19:37:36.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.506+0000 7f389cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.506+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950454932 0 0) 0x7f38a0131660 con 0x7f38a00a98f0 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.507+0000 7f389cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.507+0000 7f389cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.507+0000 7f389cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.507+0000 7f389cff9640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:37:36.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.507+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c004eb0 con 0x7f38a00a98f0 2024-03-25T19:37:36.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.511+0000 7f389cff9640 10 client.?.objecter ms_handle_connect 0x7f38a013f710 2024-03-25T19:37:36.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.511+0000 7f389cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441254178 0 0) 0x7f38a01317f0 con 0x7f38a01430c0 2024-03-25T19:37:36.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:37:36.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.516+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a0131660 con 0x7f38a01430c0 2024-03-25T19:37:36.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392581819 0 0) 0x7f38a013be70 con 0x7f38a013f710 2024-03-25T19:37:36.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:37:36.518 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.518+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a01317f0 con 0x7f38a013f710 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279100860 0 0) 0x7f388c004eb0 con 0x7f38a00a98f0 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a013be70 con 0x7f38a00a98f0 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3890001690 con 0x7f38a00a98f0 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 10 monclient(hunting): dump: 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.527 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.527+0000 7f389cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2830694082 0 0) 0x7f38a013be70 con 0x7f38a00a98f0 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 >> v1:172.21.15.29:6790/0 conn(0x7f38a01430c0 legacy=0x7f38a01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 >> v1:172.21.15.29:6789/0 conn(0x7f38a013f710 legacy=0x7f38a01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 1 monclient: found mon.b 2024-03-25T19:37:36.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0131980 con 0x7f38a00a98f0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f389cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f38a6825640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f38a6825640 10 monclient: _renew_subs 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f38a6825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.538+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a0135c60 con 0x7f38a00a98f0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.539+0000 7f38a6825640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.539+0000 7f38a6825640 10 monclient: _renew_subs 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.539+0000 7f38a6825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.539+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a00b7770 con 0x7f38a00a98f0 2024-03-25T19:37:36.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.539+0000 7f38a6825640 1 librados: init done 2024-03-25T19:37:36.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.540+0000 7f38a6825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.540+0000 7f38a6825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.540+0000 7f38a6825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:36.540 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.540+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a01371d0 con 0x7f38a00a98f0 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38900038a0 con 0x7f38a00a98f0 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3890003c70 con 0x7f38a00a98f0 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389cff9640 10 monclient: dump: 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:36.817 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:36.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.817+0000 7f389e7fc640 4 set_mon_vals no callback set 2024-03-25T19:37:36.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.818+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f3890003e90 con 0x7f38a00a98f0 2024-03-25T19:37:36.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.819+0000 7f389f7fe640 10 monclient: get_auth_request con 0x7f388c06de80 auth_method 0 2024-03-25T19:37:36.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.819+0000 7f389cff9640 10 client.4387.objecter ms_handle_connect 0x7f388c06de80 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f3890054970 con 0x7f38a00a98f0 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 10 client.4387.objecter ms_dispatch 0x7f38a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3890050480 con 0x7f38a00a98f0 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.823 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:36.823+0000 7f389cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:37.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.048+0000 7f38a6825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.048+0000 7f38a6825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.048+0000 7f38a6825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:37.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.048+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a00627b0 con 0x7f38a00a98f0 2024-03-25T19:37:37.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.071+0000 7f389cff9640 1 -- 172.21.15.29:0/3121214169 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f38900523b0 con 0x7f38a00a98f0 2024-03-25T19:37:37.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.071+0000 7f389cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.071+0000 7f389cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:37.071 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:37.071 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:37.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 10 librados: watch_flush enter 2024-03-25T19:37:37.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 10 librados: watch_flush exit 2024-03-25T19:37:37.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T19:37:37.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 10 client.4387.objecter successfully canceled tick 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 >> v1:172.21.15.29:6800/2314944989 conn(0x7f388c06de80 legacy=0x7f388c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 10 monclient: shutdown 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 >> v1:172.21.15.196:6789/0 conn(0x7f38a00a98f0 legacy=0x7f38a0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 shutdown_connections 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.073+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 >> 172.21.15.29:0/3121214169 conn(0x7f38a00b7ea0 msgr2=0x7f38a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.074+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 shutdown_connections 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.074+0000 7f38a6825640 1 -- 172.21.15.29:0/3121214169 wait complete. 2024-03-25T19:37:37.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:37.074+0000 7f38a6825640 1 librados: shutdown 2024-03-25T19:37:37.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:43.955 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.072+0000 7ff1dd02a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.072+0000 7ff1dd02a640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.072+0000 7ff1dd02a640 10 monclient: monmap: 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:44.073377+0000 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:44.073377+0000 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:44.072 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.073+0000 7ff1dd02a640 1 Processor -- start 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.073+0000 7ff1dd02a640 1 -- start start 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.073+0000 7ff1dd02a640 10 monclient: init 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d80b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d80b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d814be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80af940 con 0x7ff1d814be90 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80b0a70 con 0x7ff1d80b4d60 2024-03-25T19:37:44.074 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80b0d50 con 0x7ff1d80b1730 2024-03-25T19:37:44.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.074+0000 7ff1dd02a640 10 monclient(hunting): authenticate will time out at 6340.827451s 2024-03-25T19:37:44.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.088+0000 7ff1d6d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff1d80b4d60 0x7ff1d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41008/0 (socket says 172.21.15.29:41008) 2024-03-25T19:37:44.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.088+0000 7ff1d6d9d640 1 -- 172.21.15.29:0/3301821642 learned_addr learned my addr 172.21.15.29:0/3301821642 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:44.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234746055 0 0) 0x7ff1d80b0a70 con 0x7ff1d80b4d60 2024-03-25T19:37:44.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c0003520 con 0x7ff1d80b4d60 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738108945 0 0) 0x7ff1d80b0d50 con 0x7ff1d80b1730 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:37:44.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.099+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d80b0a70 con 0x7ff1d80b1730 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628314670 0 0) 0x7ff1d80b0a70 con 0x7ff1d80b1730 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 >> v1:172.21.15.196:6789/0 conn(0x7ff1d80b4d60 legacy=0x7ff1d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 >> v1:172.21.15.29:6789/0 conn(0x7ff1d814be90 legacy=0x7ff1d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 monclient: found mon.noname-b 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80b1030 con 0x7ff1d80b1730 2024-03-25T19:37:44.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1cc001670 con 0x7ff1d80b1730 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1d5d9b640 10 monclient: dump: 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:44.122 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:44.123 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1dd02a640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:37:44.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.121+0000 7ff1dd02a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:44.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1cc0017d0 con 0x7ff1d80b1730 2024-03-25T19:37:44.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 1 -- 172.21.15.29:0/3301821642 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1cc001ba0 con 0x7ff1d80b1730 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1dd02a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1dd02a640 4 set_mon_vals no callback set 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.135+0000 7ff1d5d9b640 10 monclient: dump: 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:44.135 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 10 monclient: shutdown 2024-03-25T19:37:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 1 -- 172.21.15.29:0/3301821642 >> v1:172.21.15.29:6790/0 conn(0x7ff1d80b1730 legacy=0x7ff1d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 1 -- 172.21.15.29:0/3301821642 shutdown_connections 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 1 -- 172.21.15.29:0/3301821642 >> 172.21.15.29:0/3301821642 conn(0x7ff1d80617a0 msgr2=0x7ff1d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 1 -- 172.21.15.29:0/3301821642 shutdown_connections 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.136+0000 7ff1dd02a640 1 -- 172.21.15.29:0/3301821642 wait complete. 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 10 monclient: monmap: 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:44.138 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 librados: starting msgr at 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 librados: starting objecter 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 Processor -- start 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 -- start start 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 librados: setting wanted keys 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 1 librados: calling monclient init 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.137+0000 7ff1dd02a640 10 monclient: init 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d80b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d80b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d814be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d8136480 con 0x7ff1d80b4d60 2024-03-25T19:37:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d81380b0 con 0x7ff1d80b1730 2024-03-25T19:37:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d81d5230 con 0x7ff1d814be90 2024-03-25T19:37:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.138+0000 7ff1dd02a640 10 monclient(hunting): authenticate will time out at 6340.891192s 2024-03-25T19:37:44.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.150+0000 7ff1d759e640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7ff1d814be90 0x7ff1d81d79d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:48702/0 (socket says 172.21.15.29:48702) 2024-03-25T19:37:44.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.150+0000 7ff1d759e640 1 -- 172.21.15.29:0/2887735008 learned_addr learned my addr 172.21.15.29:0/2887735008 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:44.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.151+0000 7ff1bf7fe640 10 client.?.objecter ms_handle_connect 0x7ff1d814be90 2024-03-25T19:37:44.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.151+0000 7ff1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.155+0000 7ff1bf7fe640 10 client.?.objecter ms_handle_connect 0x7ff1d80b1730 2024-03-25T19:37:44.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.155+0000 7ff1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.155+0000 7ff1bf7fe640 10 client.?.objecter ms_handle_connect 0x7ff1d80b4d60 2024-03-25T19:37:44.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.155+0000 7ff1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004049490 0 0) 0x7ff1d8136480 con 0x7ff1d80b4d60 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:37:44.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.162+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4004eb0 con 0x7ff1d80b4d60 2024-03-25T19:37:44.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526671911 0 0) 0x7ff1d81d5230 con 0x7ff1d814be90 2024-03-25T19:37:44.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:37:44.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.163+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d8136480 con 0x7ff1d814be90 2024-03-25T19:37:44.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722988220 0 0) 0x7ff1d81380b0 con 0x7ff1d80b1730 2024-03-25T19:37:44.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:37:44.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.169+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d81d5230 con 0x7ff1d80b1730 2024-03-25T19:37:44.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 629481867 0 0) 0x7ff1d8136480 con 0x7ff1d814be90 2024-03-25T19:37:44.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1d81380b0 con 0x7ff1d814be90 2024-03-25T19:37:44.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1d0001cb0 con 0x7ff1d814be90 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.174+0000 7ff1bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:44.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711097935 0 0) 0x7ff1c4004eb0 con 0x7ff1d80b4d60 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1d8136480 con 0x7ff1d80b4d60 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1cc001ae0 con 0x7ff1d80b4d60 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.178+0000 7ff1bf7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:44.178 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:44.179 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4050094980 0 0) 0x7ff1d81380b0 con 0x7ff1d814be90 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 >> v1:172.21.15.196:6789/0 conn(0x7ff1d80b1730 legacy=0x7ff1d8135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 >> v1:172.21.15.29:6789/0 conn(0x7ff1d80b4d60 legacy=0x7ff1d812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 1 monclient: found mon.c 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d8136e70 con 0x7ff1d814be90 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 10 monclient: _renew_subs 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d8130400 con 0x7ff1d814be90 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 10 monclient: _renew_subs 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81d80e0 con 0x7ff1d814be90 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.184+0000 7ff1dd02a640 1 librados: init done 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.185+0000 7ff1dd02a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.185+0000 7ff1dd02a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.185+0000 7ff1dd02a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.185+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d8137f20 con 0x7ff1d814be90 2024-03-25T19:37:44.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1d0003990 con 0x7ff1d814be90 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1d0003da0 con 0x7ff1d814be90 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 10 monclient: dump: 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:44.512 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1d559a640 4 set_mon_vals no callback set 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.511+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ff1d0003fc0 con 0x7ff1d814be90 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.512+0000 7ff1d6d9d640 10 monclient: get_auth_request con 0x7ff1c406df60 auth_method 0 2024-03-25T19:37:44.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.512+0000 7ff1bf7fe640 10 client.4394.objecter ms_handle_connect 0x7ff1c406df60 2024-03-25T19:37:44.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ff1d0054ac0 con 0x7ff1d814be90 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 10 client.4394.objecter ms_dispatch 0x7ff1d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1d00505d0 con 0x7ff1d814be90 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.520+0000 7ff1bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:44.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.781+0000 7ff1dd02a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.781+0000 7ff1dd02a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.781+0000 7ff1dd02a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:37:44.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.781+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1d80627b0 con 0x7ff1d814be90 2024-03-25T19:37:44.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.810+0000 7ff1bf7fe640 1 -- 172.21.15.29:0/2887735008 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ff1d0052500 con 0x7ff1d814be90 2024-03-25T19:37:44.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.810+0000 7ff1bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.810+0000 7ff1bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:44.810 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:44.810 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:44.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 10 librados: watch_flush enter 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 10 librados: watch_flush exit 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 >> v1:172.21.15.29:6800/2314944989 conn(0x7ff1c406df60 legacy=0x7ff1c4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 10 monclient: shutdown 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.812+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 >> v1:172.21.15.29:6790/0 conn(0x7ff1d814be90 legacy=0x7ff1d81d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.813+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 shutdown_connections 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.813+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 >> 172.21.15.29:0/2887735008 conn(0x7ff1d80617a0 msgr2=0x7ff1d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.813+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 shutdown_connections 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.813+0000 7ff1dd02a640 1 -- 172.21.15.29:0/2887735008 wait complete. 2024-03-25T19:37:44.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:44.813+0000 7ff1dd02a640 1 librados: shutdown 2024-03-25T19:37:44.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:51.631 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.747+0000 7efca5e0e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.747+0000 7efca5e0e640 10 monclient: build_initial_monmap 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.747+0000 7efca5e0e640 10 monclient: monmap: 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:51.748200+0000 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:51.748200+0000 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:51.747 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:51.748 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:51.748 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.748+0000 7efca5e0e640 1 Processor -- start 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.748+0000 7efca5e0e640 1 -- start start 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.748+0000 7efca5e0e640 10 monclient: init 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-b con 0x7efca00705b0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-c con 0x7efca01402d0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-a con 0x7efca0149ae0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00718c0 con 0x7efca0149ae0 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00714c0 con 0x7efca01402d0 2024-03-25T19:37:51.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0072b50 con 0x7efca00705b0 2024-03-25T19:37:51.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:51.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.749+0000 7efca5e0e640 10 monclient(hunting): authenticate will time out at 6348.502385s 2024-03-25T19:37:51.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.759+0000 7efc9effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7efca01402d0 0x7efca0140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:60462/0 (socket says 172.21.15.29:60462) 2024-03-25T19:37:51.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.759+0000 7efc9effd640 1 -- 172.21.15.29:0/2897629518 learned_addr learned my addr 172.21.15.29:0/2897629518 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:51.770 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463991078 0 0) 0x7efca0072b50 con 0x7efca00705b0 2024-03-25T19:37:51.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:37:51.771 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.771+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88003520 con 0x7efca00705b0 2024-03-25T19:37:51.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414620059 0 0) 0x7efca00718c0 con 0x7efca0149ae0 2024-03-25T19:37:51.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0072b50 con 0x7efca0149ae0 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737802348 0 0) 0x7efca00714c0 con 0x7efca01402d0 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:37:51.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.776+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca00718c0 con 0x7efca01402d0 2024-03-25T19:37:51.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913308094 0 0) 0x7efc88003520 con 0x7efca00705b0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 >> v1:172.21.15.196:6789/0 conn(0x7efca01402d0 legacy=0x7efca0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 >> v1:172.21.15.29:6789/0 conn(0x7efca0149ae0 legacy=0x7efca014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca013dd10 con 0x7efca00705b0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efc90001690 con 0x7efca00705b0 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efc9e7fc640 10 monclient: dump: 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efca5e0e640 5 monclient: authenticate success, global_id 4397 2024-03-25T19:37:51.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.792+0000 7efca5e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc900017f0 con 0x7efca00705b0 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 1 -- 172.21.15.29:0/2897629518 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efc90001c00 con 0x7efca00705b0 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efca5e0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efca5e0e640 4 set_mon_vals no callback set 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.803+0000 7efc9e7fc640 10 monclient: dump: 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:51.803 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:51.804 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 10 monclient: shutdown 2024-03-25T19:37:51.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:51.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 1 -- 172.21.15.29:0/2897629518 >> v1:172.21.15.29:6790/0 conn(0x7efca00705b0 legacy=0x7efca013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 1 -- 172.21.15.29:0/2897629518 shutdown_connections 2024-03-25T19:37:51.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 1 -- 172.21.15.29:0/2897629518 >> 172.21.15.29:0/2897629518 conn(0x7efca00b5d10 msgr2=0x7efca00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 1 -- 172.21.15.29:0/2897629518 shutdown_connections 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.804+0000 7efca5e0e640 1 -- 172.21.15.29:0/2897629518 wait complete. 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.805+0000 7efca5e0e640 10 monclient: build_initial_monmap 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.805+0000 7efca5e0e640 10 monclient: monmap: 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.805+0000 7efca5e0e640 1 librados: starting msgr at 2024-03-25T19:37:51.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.805+0000 7efca5e0e640 1 librados: starting objecter 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.805+0000 7efca5e0e640 1 Processor -- start 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 1 -- start start 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 1 librados: setting wanted keys 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 1 librados: calling monclient init 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 10 monclient: init 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:51.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-b con 0x7efca00705b0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.806+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-c con 0x7efca01402d0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 10 monclient(hunting): picked mon.noname-a con 0x7efca0149ae0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca01d5420 con 0x7efca0149ae0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca01420d0 con 0x7efca00705b0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca013fd60 con 0x7efca01402d0 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:51.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.807+0000 7efca5e0e640 10 monclient(hunting): authenticate will time out at 6348.559972s 2024-03-25T19:37:51.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.814+0000 7efc9f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7efca00705b0 0x7efca013f650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:60504/0 (socket says 172.21.15.29:60504) 2024-03-25T19:37:51.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.814+0000 7efc9f7fe640 1 -- 172.21.15.29:0/3759199663 learned_addr learned my addr 172.21.15.29:0/3759199663 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:51.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.816+0000 7efc87fff640 10 client.?.objecter ms_handle_connect 0x7efca00705b0 2024-03-25T19:37:51.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.816+0000 7efc87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.821+0000 7efc87fff640 10 client.?.objecter ms_handle_connect 0x7efca01402d0 2024-03-25T19:37:51.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.821+0000 7efc87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.830+0000 7efc87fff640 10 client.?.objecter ms_handle_connect 0x7efca0149ae0 2024-03-25T19:37:51.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.830+0000 7efc87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435143448 0 0) 0x7efca01d5420 con 0x7efca0149ae0 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:37:51.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.839+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c004eb0 con 0x7efca0149ae0 2024-03-25T19:37:51.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338952262 0 0) 0x7efca01420d0 con 0x7efca00705b0 2024-03-25T19:37:51.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:37:51.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.840+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca01d5420 con 0x7efca00705b0 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280689021 0 0) 0x7efca013fd60 con 0x7efca01402d0 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:37:51.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.846+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca01420d0 con 0x7efca01402d0 2024-03-25T19:37:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350156320 0 0) 0x7efca01d5420 con 0x7efca00705b0 2024-03-25T19:37:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca013fd60 con 0x7efca00705b0 2024-03-25T19:37:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efc90001b80 con 0x7efca00705b0 2024-03-25T19:37:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 10 monclient(hunting): dump: 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.861+0000 7efc87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925384946 0 0) 0x7efc8c004eb0 con 0x7efca0149ae0 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca01d5420 con 0x7efca0149ae0 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efc98001bd0 con 0x7efca0149ae0 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.862+0000 7efc87fff640 10 monclient(hunting): dump: 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:51.863 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:51.864 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2509041956 0 0) 0x7efca013fd60 con 0x7efca00705b0 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 >> v1:172.21.15.29:6790/0 conn(0x7efca01402d0 legacy=0x7efca01d5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 >> v1:172.21.15.29:6789/0 conn(0x7efca0149ae0 legacy=0x7efca0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 1 monclient: found mon.b 2024-03-25T19:37:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca013fef0 con 0x7efca00705b0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efc87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 10 monclient: _renew_subs 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca0132ba0 con 0x7efca00705b0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 10 monclient: _renew_subs 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efca013eae0 con 0x7efca00705b0 2024-03-25T19:37:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.869+0000 7efca5e0e640 1 librados: init done 2024-03-25T19:37:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.870+0000 7efca5e0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.871+0000 7efca5e0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.871+0000 7efca5e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:51.871+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca01d5610 con 0x7efca00705b0 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc90001230 con 0x7efca00705b0 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efc90003a40 con 0x7efca00705b0 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc87fff640 10 monclient: dump: 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:52.156 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:52.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.156+0000 7efc9dffb640 4 set_mon_vals no callback set 2024-03-25T19:37:52.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.157+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7efc90003c60 con 0x7efca00705b0 2024-03-25T19:37:52.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.157+0000 7efc9effd640 10 monclient: get_auth_request con 0x7efc8c06df70 auth_method 0 2024-03-25T19:37:52.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.158+0000 7efc87fff640 10 client.4399.objecter ms_handle_connect 0x7efc8c06df70 2024-03-25T19:37:52.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.163+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7efc900548d0 con 0x7efca00705b0 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.163+0000 7efc87fff640 10 client.4399.objecter ms_dispatch 0x7efca0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.163+0000 7efc87fff640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.163+0000 7efc87fff640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.163+0000 7efc87fff640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.164+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc90050210 con 0x7efca00705b0 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.164+0000 7efc87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.164+0000 7efc87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:52.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.421+0000 7efca5e0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.421+0000 7efca5e0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.421+0000 7efca5e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:52.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.421+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efca00627b0 con 0x7efca00705b0 2024-03-25T19:37:52.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.472+0000 7efc87fff640 1 -- 172.21.15.29:0/3759199663 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7efc90052310 con 0x7efca00705b0 2024-03-25T19:37:52.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.472+0000 7efc87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.472+0000 7efc87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:52.472 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:37:52.472 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:52.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 10 librados: watch_flush enter 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 10 librados: watch_flush exit 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 10 client.4399.objecter successfully canceled tick 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 >> v1:172.21.15.29:6800/2314944989 conn(0x7efc8c06df70 legacy=0x7efc8c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 10 monclient: shutdown 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 >> v1:172.21.15.196:6789/0 conn(0x7efca00705b0 legacy=0x7efca013f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 shutdown_connections 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.474+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 >> 172.21.15.29:0/3759199663 conn(0x7efca00b5d10 msgr2=0x7efca013c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.475+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 shutdown_connections 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.475+0000 7efca5e0e640 1 -- 172.21.15.29:0/3759199663 wait complete. 2024-03-25T19:37:52.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:52.475+0000 7efca5e0e640 1 librados: shutdown 2024-03-25T19:37:52.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:59.292 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.409+0000 7fbbf02ff640 10 monclient: get_monmap_and_config 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.409+0000 7fbbf02ff640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.409+0000 7fbbf02ff640 10 monclient: monmap: 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:37:59.410180+0000 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:37:59.410180+0000 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:37:59.409 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.410+0000 7fbbf02ff640 1 Processor -- start 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.410+0000 7fbbf02ff640 1 -- start start 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.410+0000 7fbbf02ff640 10 monclient: init 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe80a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe81430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8142520 con 0x7fbbe813f710 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80b28a0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80b2a30 con 0x7fbbe81430c0 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.411+0000 7fbbf02ff640 10 monclient(hunting): authenticate will time out at 6356.164187s 2024-03-25T19:37:59.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.419+0000 7fbbed89a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbbe813f710 0x7fbbe81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33712/0 (socket says 172.21.15.29:33712) 2024-03-25T19:37:59.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.419+0000 7fbbed89a640 1 -- 172.21.15.29:0/4203813621 learned_addr learned my addr 172.21.15.29:0/4203813621 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:59.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456499584 0 0) 0x7fbbe8142520 con 0x7fbbe813f710 2024-03-25T19:37:59.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:37:59.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.437+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd4003520 con 0x7fbbe813f710 2024-03-25T19:37:59.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334410395 0 0) 0x7fbbe80b28a0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:37:59.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.438+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe8142520 con 0x7fbbe80a98f0 2024-03-25T19:37:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955750759 0 0) 0x7fbbe8142520 con 0x7fbbe80a98f0 2024-03-25T19:37:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 >> v1:172.21.15.29:6790/0 conn(0x7fbbe81430c0 legacy=0x7fbbe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 >> v1:172.21.15.29:6789/0 conn(0x7fbbe813f710 legacy=0x7fbbe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 monclient: found mon.noname-c 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80b44c0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbe4001670 con 0x7fbbe80a98f0 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbed099640 10 monclient: dump: 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:59.460 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbf02ff640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:37:59.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.460+0000 7fbbf02ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe40017d0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 1 -- 172.21.15.29:0/4203813621 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbe4001be0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbed099640 10 monclient: dump: 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.467 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbf02ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:59.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.467+0000 7fbbf02ff640 4 set_mon_vals no callback set 2024-03-25T19:37:59.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.468+0000 7fbbf02ff640 10 monclient: shutdown 2024-03-25T19:37:59.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.468+0000 7fbbf02ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.468+0000 7fbbf02ff640 1 -- 172.21.15.29:0/4203813621 >> v1:172.21.15.196:6789/0 conn(0x7fbbe80a98f0 legacy=0x7fbbe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.468+0000 7fbbf02ff640 1 -- 172.21.15.29:0/4203813621 shutdown_connections 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.468+0000 7fbbf02ff640 1 -- 172.21.15.29:0/4203813621 >> 172.21.15.29:0/4203813621 conn(0x7fbbe80b7ea0 msgr2=0x7fbbe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.469+0000 7fbbf02ff640 1 -- 172.21.15.29:0/4203813621 shutdown_connections 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.469+0000 7fbbf02ff640 1 -- 172.21.15.29:0/4203813621 wait complete. 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.469+0000 7fbbf02ff640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.469+0000 7fbbf02ff640 10 monclient: monmap: 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:37:59.470 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 librados: starting msgr at 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 librados: starting objecter 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 Processor -- start 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 -- start start 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 librados: setting wanted keys 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 librados: calling monclient init 2024-03-25T19:37:59.471 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient: init 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe80a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe81430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80b7c10 con 0x7fbbe80a98f0 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.470+0000 7fbbf02ff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8131660 con 0x7fbbe813f710 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.471+0000 7fbbf02ff640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe81317f0 con 0x7fbbe81430c0 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.471+0000 7fbbf02ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.471+0000 7fbbf02ff640 10 monclient(hunting): authenticate will time out at 6356.223882s 2024-03-25T19:37:59.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.483+0000 7fbbed89a640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbbe813f710 0x7fbbe81dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:49382/0 (socket says 172.21.15.29:49382) 2024-03-25T19:37:59.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.483+0000 7fbbed89a640 1 -- 172.21.15.29:0/546184644 learned_addr learned my addr 172.21.15.29:0/546184644 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:37:59.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.485+0000 7fbbdeffd640 10 client.?.objecter ms_handle_connect 0x7fbbe813f710 2024-03-25T19:37:59.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.485+0000 7fbbdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957021325 0 0) 0x7fbbe8131660 con 0x7fbbe813f710 2024-03-25T19:37:59.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:37:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.491+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc004eb0 con 0x7fbbe813f710 2024-03-25T19:37:59.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.496+0000 7fbbdeffd640 10 client.?.objecter ms_handle_connect 0x7fbbe80a98f0 2024-03-25T19:37:59.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.496+0000 7fbbdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.502+0000 7fbbdeffd640 10 client.?.objecter ms_handle_connect 0x7fbbe81430c0 2024-03-25T19:37:59.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.502+0000 7fbbdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616006929 0 0) 0x7fbbcc004eb0 con 0x7fbbe813f710 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbe8131660 con 0x7fbbe813f710 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbd8001990 con 0x7fbbe813f710 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 10 monclient(hunting): dump: 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.510 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.510+0000 7fbbdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.511+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742066593 0 0) 0x7fbbe80b7c10 con 0x7fbbe80a98f0 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.512+0000 7fbbdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.512+0000 7fbbdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.512+0000 7fbbdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.512+0000 7fbbdeffd640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:37:59.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.512+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc004eb0 con 0x7fbbe80a98f0 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.514+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114073211 0 0) 0x7fbbe81317f0 con 0x7fbbe81430c0 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.514+0000 7fbbdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.515+0000 7fbbdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.515+0000 7fbbdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.515+0000 7fbbdeffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:37:59.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.515+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe80b7c10 con 0x7fbbe81430c0 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 886348731 0 0) 0x7fbbe8131660 con 0x7fbbe813f710 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 >> v1:172.21.15.29:6790/0 conn(0x7fbbe81430c0 legacy=0x7fbbe81e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 >> v1:172.21.15.29:6789/0 conn(0x7fbbe80a98f0 legacy=0x7fbbe8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 1 monclient: found mon.b 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8131980 con 0x7fbbe813f710 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 10 monclient: _renew_subs 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe8135070 con 0x7fbbe813f710 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 10 monclient: _renew_subs 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe813b460 con 0x7fbbe813f710 2024-03-25T19:37:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.519+0000 7fbbf02ff640 1 librados: init done 2024-03-25T19:37:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.520+0000 7fbbf02ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.520+0000 7fbbf02ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.520+0000 7fbbf02ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:37:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.520+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe80627b0 con 0x7fbbe813f710 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbd80047e0 con 0x7fbbe813f710 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbd8004bb0 con 0x7fbbe813f710 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbdeffd640 10 monclient: dump: 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:37:59.791 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:37:59.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.791+0000 7fbbec898640 4 set_mon_vals no callback set 2024-03-25T19:37:59.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.792+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbbd8004dd0 con 0x7fbbe813f710 2024-03-25T19:37:59.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.793+0000 7fbbee09b640 10 monclient: get_auth_request con 0x7fbbcc06de80 auth_method 0 2024-03-25T19:37:59.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.793+0000 7fbbdeffd640 10 client.4405.objecter ms_handle_connect 0x7fbbcc06de80 2024-03-25T19:37:59.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.799+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fbbd80548d0 con 0x7fbbe813f710 2024-03-25T19:37:59.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.799+0000 7fbbdeffd640 10 client.4405.objecter ms_dispatch 0x7fbbe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:59.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.799+0000 7fbbdeffd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:59.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.799+0000 7fbbdeffd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:59.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.799+0000 7fbbdeffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T19:37:59.799 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.800+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbd80503e0 con 0x7fbbe813f710 2024-03-25T19:37:59.799 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.800+0000 7fbbdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:37:59.800+0000 7fbbdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:00.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.033+0000 7fbbf02ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.033+0000 7fbbf02ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.033+0000 7fbbf02ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:00.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.033+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbe813b760 con 0x7fbbe813f710 2024-03-25T19:38:00.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.064+0000 7fbbdeffd640 1 -- 172.21.15.29:0/546184644 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fbbd8052310 con 0x7fbbe813f710 2024-03-25T19:38:00.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.064+0000 7fbbdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.064+0000 7fbbdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:00.064 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:00.064 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 10 librados: watch_flush enter 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 10 librados: watch_flush exit 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 10 client.4405.objecter successfully canceled tick 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbbcc06de80 legacy=0x7fbbcc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 10 monclient: shutdown 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 >> v1:172.21.15.196:6789/0 conn(0x7fbbe813f710 legacy=0x7fbbe81dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 shutdown_connections 2024-03-25T19:38:00.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.066+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 >> 172.21.15.29:0/546184644 conn(0x7fbbe80b7ea0 msgr2=0x7fbbe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:00.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.067+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 shutdown_connections 2024-03-25T19:38:00.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.067+0000 7fbbf02ff640 1 -- 172.21.15.29:0/546184644 wait complete. 2024-03-25T19:38:00.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:00.067+0000 7fbbf02ff640 1 librados: shutdown 2024-03-25T19:38:00.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:04.997 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:35.161 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.050 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:35.214 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.096 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:35.260 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.105 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:35.267 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.110 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:35.272 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.112 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:35.276 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.121 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:35.284 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.147 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:35.309 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:05.151 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:35.315 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.885 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:07.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.001+0000 7f206a789640 10 monclient: get_monmap_and_config 2024-03-25T19:38:07.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.001+0000 7f206a789640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.001+0000 7f206a789640 10 monclient: monmap: 2024-03-25T19:38:07.000 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:07.002106+0000 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:07.002106+0000 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:07.001 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 1 Processor -- start 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 1 -- start start 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient: init 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:07.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient(hunting): picked mon.noname-b con 0x7f20640a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient(hunting): picked mon.noname-a con 0x7f206413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 10 monclient(hunting): picked mon.noname-c con 0x7f20641430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.002+0000 7f206a789640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2064142520 con 0x7f206413f710 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.003+0000 7f206a789640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640b28a0 con 0x7f20641430c0 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.003+0000 7f206a789640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640b2a30 con 0x7f20640a98f0 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.003+0000 7f206a789640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.003+0000 7f206a789640 10 monclient(hunting): authenticate will time out at 6363.755916s 2024-03-25T19:38:07.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.010+0000 7f2068d26640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f20641430c0 0x7f206414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:54830/0 (socket says 172.21.15.29:54830) 2024-03-25T19:38:07.009 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.010+0000 7f2068d26640 1 -- 172.21.15.29:0/2524060273 learned_addr learned my addr 172.21.15.29:0/2524060273 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.028+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125346991 0 0) 0x7f20640b28a0 con 0x7f20641430c0 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.029+0000 7f2062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.029+0000 7f2062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.029+0000 7f2062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.029+0000 7f2062ffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:38:07.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.029+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f204c003520 con 0x7f20641430c0 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.030+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692337653 0 0) 0x7f20640b2a30 con 0x7f20640a98f0 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.030+0000 7f2062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.030+0000 7f2062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.030+0000 7f2062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.030+0000 7f2062ffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:38:07.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.031+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20640b28a0 con 0x7f20640a98f0 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588225791 0 0) 0x7f2064142520 con 0x7f206413f710 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:38:07.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.034+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20640b2a30 con 0x7f206413f710 2024-03-25T19:38:07.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081322922 0 0) 0x7f20640b2a30 con 0x7f206413f710 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 >> v1:172.21.15.29:6790/0 conn(0x7f20640a98f0 legacy=0x7f20640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 >> v1:172.21.15.196:6789/0 conn(0x7f20641430c0 legacy=0x7f206414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 monclient: found mon.noname-a 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640b44c0 con 0x7f206413f710 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f20580015b0 con 0x7f206413f710 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f2062ffd640 10 monclient: dump: 2024-03-25T19:38:07.046 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f206a789640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.046+0000 7f206a789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:07.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.048+0000 7f2062ffd640 1 -- 172.21.15.29:0/2524060273 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2058001710 con 0x7f206413f710 2024-03-25T19:38:07.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.048+0000 7f2062ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.048+0000 7f206a789640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:07.048 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.048+0000 7f206a789640 4 set_mon_vals no callback set 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 10 monclient: shutdown 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 1 -- 172.21.15.29:0/2524060273 >> v1:172.21.15.29:6789/0 conn(0x7f206413f710 legacy=0x7f20641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 1 -- 172.21.15.29:0/2524060273 shutdown_connections 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 1 -- 172.21.15.29:0/2524060273 >> 172.21.15.29:0/2524060273 conn(0x7f20640b7ea0 msgr2=0x7f20640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 1 -- 172.21.15.29:0/2524060273 shutdown_connections 2024-03-25T19:38:07.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.049+0000 7f206a789640 1 -- 172.21.15.29:0/2524060273 wait complete. 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.050+0000 7f206a789640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.050+0000 7f206a789640 10 monclient: monmap: 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:07.050 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.050+0000 7f206a789640 1 librados: starting msgr at 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.050+0000 7f206a789640 1 librados: starting objecter 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 1 Processor -- start 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 1 -- start start 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 1 librados: setting wanted keys 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 1 librados: calling monclient init 2024-03-25T19:38:07.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 10 monclient: init 2024-03-25T19:38:07.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.051+0000 7f206a789640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 10 monclient(hunting): picked mon.noname-c con 0x7f20640a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 10 monclient(hunting): picked mon.noname-a con 0x7f206413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 10 monclient(hunting): picked mon.noname-b con 0x7f20641430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20641dba70 con 0x7f206413f710 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640b6ca0 con 0x7f20641430c0 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206413b0a0 con 0x7f20640a98f0 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.053 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.052+0000 7f206a789640 10 monclient(hunting): authenticate will time out at 6363.805084s 2024-03-25T19:38:07.059 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.060+0000 7f2068d26640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f20641430c0 0x7f20641db360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:54862/0 (socket says 172.21.15.29:54862) 2024-03-25T19:38:07.059 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.060+0000 7f2068d26640 1 -- 172.21.15.29:0/1538687285 learned_addr learned my addr 172.21.15.29:0/1538687285 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:07.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.063+0000 7f2060ff9640 10 client.?.objecter ms_handle_connect 0x7f20641430c0 2024-03-25T19:38:07.062 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.063+0000 7f2060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018349865 0 0) 0x7f20640b6ca0 con 0x7f20641430c0 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:38:07.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.068+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2050004eb0 con 0x7f20641430c0 2024-03-25T19:38:07.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.071+0000 7f2060ff9640 10 client.?.objecter ms_handle_connect 0x7f20640a98f0 2024-03-25T19:38:07.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.071+0000 7f2060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973556243 0 0) 0x7f206413b0a0 con 0x7f20640a98f0 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:38:07.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.079+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20640b6ca0 con 0x7f20640a98f0 2024-03-25T19:38:07.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.085+0000 7f2060ff9640 10 client.?.objecter ms_handle_connect 0x7f206413f710 2024-03-25T19:38:07.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.085+0000 7f2060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.090 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358239774 0 0) 0x7f2050004eb0 con 0x7f20641430c0 2024-03-25T19:38:07.090 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f206413b0a0 con 0x7f20641430c0 2024-03-25T19:38:07.090 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f205c001b70 con 0x7f20641430c0 2024-03-25T19:38:07.090 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.091+0000 7f2060ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973657231 0 0) 0x7f20641dba70 con 0x7f206413f710 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:38:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.092+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2050004eb0 con 0x7f206413f710 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519025324 0 0) 0x7f20640b6ca0 con 0x7f20640a98f0 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20641dba70 con 0x7f20640a98f0 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2054001930 con 0x7f20640a98f0 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.095+0000 7f2060ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:07.095 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:07.096 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:07.096 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:07.096 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968395777 0 0) 0x7f206413b0a0 con 0x7f20641430c0 2024-03-25T19:38:07.103 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 >> v1:172.21.15.29:6790/0 conn(0x7f20640a98f0 legacy=0x7f2064139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 >> v1:172.21.15.29:6789/0 conn(0x7f206413f710 legacy=0x7f20641dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 1 monclient: found mon.b 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206413b380 con 0x7f20641430c0 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.104 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f2060ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 10 monclient: _renew_subs 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2064139990 con 0x7f20641430c0 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 10 monclient: _renew_subs 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20640b79b0 con 0x7f20641430c0 2024-03-25T19:38:07.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.104+0000 7f206a789640 1 librados: init done 2024-03-25T19:38:07.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.105+0000 7f206a789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.105+0000 7f206a789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.105+0000 7f206a789640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:07.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.105+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206414f2d0 con 0x7f20641430c0 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f205c002060 con 0x7f20641430c0 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f205c004d00 con 0x7f20641430c0 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f2060ff9640 10 monclient: dump: 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:07.369 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.369+0000 7f20627fc640 4 set_mon_vals no callback set 2024-03-25T19:38:07.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.370+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f205c01acd0 con 0x7f20641430c0 2024-03-25T19:38:07.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.371+0000 7f2063fff640 10 monclient: get_auth_request con 0x7f205006df20 auth_method 0 2024-03-25T19:38:07.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.371+0000 7f2060ff9640 10 client.4411.objecter ms_handle_connect 0x7f205006df20 2024-03-25T19:38:07.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.379+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f205c054af0 con 0x7f20641430c0 2024-03-25T19:38:07.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.379+0000 7f2060ff9640 10 client.4411.objecter ms_dispatch 0x7f2064000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:07.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.379+0000 7f2060ff9640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:07.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.379+0000 7f2060ff9640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:07.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.379+0000 7f2060ff9640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T19:38:07.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.380+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f205c050430 con 0x7f20641430c0 2024-03-25T19:38:07.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.380+0000 7f2060ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.380+0000 7f2060ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:07.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.630+0000 7f206a789640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.630+0000 7f206a789640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.630+0000 7f206a789640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:07.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.630+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20640627b0 con 0x7f20641430c0 2024-03-25T19:38:07.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.654+0000 7f2060ff9640 1 -- 172.21.15.29:0/1538687285 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f205c052530 con 0x7f20641430c0 2024-03-25T19:38:07.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.654+0000 7f2060ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.654+0000 7f2060ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:07.654 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:07.654 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 10 librados: watch_flush enter 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 10 librados: watch_flush exit 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 10 client.4411.objecter successfully canceled tick 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 >> v1:172.21.15.29:6800/2314944989 conn(0x7f205006df20 legacy=0x7f20500703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 10 monclient: shutdown 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 >> v1:172.21.15.196:6789/0 conn(0x7f20641430c0 legacy=0x7f20641db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 shutdown_connections 2024-03-25T19:38:07.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 >> 172.21.15.29:0/1538687285 conn(0x7f20640b7ea0 msgr2=0x7f2064074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:07.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 shutdown_connections 2024-03-25T19:38:07.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 -- 172.21.15.29:0/1538687285 wait complete. 2024-03-25T19:38:07.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:07.657+0000 7f206a789640 1 librados: shutdown 2024-03-25T19:38:07.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:14.475 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:14.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.586+0000 7fbc13b82640 10 monclient: get_monmap_and_config 2024-03-25T19:38:14.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.586+0000 7fbc13b82640 10 monclient: build_initial_monmap 2024-03-25T19:38:14.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.586+0000 7fbc13b82640 10 monclient: monmap: 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:14.587859+0000 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:14.587859+0000 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:14.587 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.586+0000 7fbc13b82640 1 Processor -- start 2024-03-25T19:38:14.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 1 -- start start 2024-03-25T19:38:14.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient: init 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.587+0000 7fbc13b82640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0af940 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.588+0000 7fbc13b82640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0b0a70 con 0x7fbc0c0b1730 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.588+0000 7fbc13b82640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0b0d50 con 0x7fbc0c14be90 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.588+0000 7fbc13b82640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:14.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.588+0000 7fbc13b82640 10 monclient(hunting): authenticate will time out at 6371.341882s 2024-03-25T19:38:14.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.592+0000 7fbc1111d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbc0c0b1730 0x7fbc0c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:59486/0 (socket says 172.21.15.29:59486) 2024-03-25T19:38:14.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.592+0000 7fbc1111d640 1 -- 172.21.15.29:0/3092923499 learned_addr learned my addr 172.21.15.29:0/3092923499 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:14.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813304512 0 0) 0x7fbc0c0b0a70 con 0x7fbc0c0b1730 2024-03-25T19:38:14.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:38:14.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.602+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8003520 con 0x7fbc0c0b1730 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478414423 0 0) 0x7fbc0c0b0d50 con 0x7fbc0c14be90 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:38:14.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.605+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c0b0a70 con 0x7fbc0c14be90 2024-03-25T19:38:14.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068066317 0 0) 0x7fbc0c0af940 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:38:14.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.607+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c0b0d50 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289112791 0 0) 0x7fbbf8003520 con 0x7fbc0c0b1730 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 >> v1:172.21.15.29:6790/0 conn(0x7fbc0c14be90 legacy=0x7fbc0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 >> v1:172.21.15.29:6789/0 conn(0x7fbc0c0b4d60 legacy=0x7fbc0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: _finish_hunting 0 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 monclient: found mon.noname-c 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0b1030 con 0x7fbc0c0b1730 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: _finish_auth 0 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: _check_auth_tickets 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbfc001690 con 0x7fbc0c0b1730 2024-03-25T19:38:14.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc13b82640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc13b82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc1091c640 10 monclient: dump: 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.613+0000 7fbc13b82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc0017f0 con 0x7fbc0c0b1730 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 1 -- 172.21.15.29:0/3092923499 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbfc001c00 con 0x7fbc0c0b1730 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc13b82640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc13b82640 4 set_mon_vals no callback set 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.624+0000 7fbc1091c640 10 monclient: dump: 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:14.626 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 10 monclient: shutdown 2024-03-25T19:38:14.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:14.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 1 -- 172.21.15.29:0/3092923499 >> v1:172.21.15.196:6789/0 conn(0x7fbc0c0b1730 legacy=0x7fbc0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 1 -- 172.21.15.29:0/3092923499 shutdown_connections 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 1 -- 172.21.15.29:0/3092923499 >> 172.21.15.29:0/3092923499 conn(0x7fbc0c0617a0 msgr2=0x7fbc0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 1 -- 172.21.15.29:0/3092923499 shutdown_connections 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.625+0000 7fbc13b82640 1 -- 172.21.15.29:0/3092923499 wait complete. 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.626+0000 7fbc13b82640 10 monclient: build_initial_monmap 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.626+0000 7fbc13b82640 10 monclient: monmap: 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:14.628 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 librados: starting msgr at 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 librados: starting objecter 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 Processor -- start 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 -- start start 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 librados: setting wanted keys 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 1 librados: calling monclient init 2024-03-25T19:38:14.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 10 monclient: init 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.627+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0c0b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c14be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c141790 con 0x7fbc0c14be90 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c141a00 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c1d7a90 con 0x7fbc0c0b1730 2024-03-25T19:38:14.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:14.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.628+0000 7fbc13b82640 10 monclient(hunting): authenticate will time out at 6371.382098s 2024-03-25T19:38:14.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.634+0000 7fbc1191e640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fbc0c0b1730 0x7fbc0c13b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:48978/0 (socket says 172.21.15.29:48978) 2024-03-25T19:38:14.635 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.634+0000 7fbc1191e640 1 -- 172.21.15.29:0/3501834601 learned_addr learned my addr 172.21.15.29:0/3501834601 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:14.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.636+0000 7fbc027fc640 10 client.?.objecter ms_handle_connect 0x7fbc0c0b1730 2024-03-25T19:38:14.636 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.636+0000 7fbc027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.636+0000 7fbc027fc640 10 client.?.objecter ms_handle_connect 0x7fbc0c0b4d60 2024-03-25T19:38:14.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.636+0000 7fbc027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.645 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956357499 0 0) 0x7fbc0c1d7a90 con 0x7fbc0c0b1730 2024-03-25T19:38:14.645 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.645 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.645 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.645 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:38:14.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.645+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc0c0b1730 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042515717 0 0) 0x7fbc0c141a00 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:38:14.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.646+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c1d7a90 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.651+0000 7fbc027fc640 10 client.?.objecter ms_handle_connect 0x7fbc0c14be90 2024-03-25T19:38:14.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.651+0000 7fbc027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896325071 0 0) 0x7fbc0c1d7a90 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc0c141a00 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbbfc001b00 con 0x7fbc0c0b4d60 2024-03-25T19:38:14.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 10 monclient(hunting): dump: 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.660 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:14.661 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:14.661 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:14.661 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.661 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.659+0000 7fbc027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 867696636 0 0) 0x7fbbf0004eb0 con 0x7fbc0c0b1730 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc0c1d7a90 con 0x7fbc0c0b1730 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbc08001a10 con 0x7fbc0c0b1730 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:14.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.664+0000 7fbc027fc640 10 monclient(hunting): dump: 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:14.665 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762539907 0 0) 0x7fbc0c141790 con 0x7fbc0c14be90 2024-03-25T19:38:14.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:38:14.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.669+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc0c14be90 2024-03-25T19:38:14.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 286202980 0 0) 0x7fbc0c1d7a90 con 0x7fbc0c0b1730 2024-03-25T19:38:14.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 >> v1:172.21.15.196:6789/0 conn(0x7fbc0c0b4d60 legacy=0x7fbc0c142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 >> v1:172.21.15.29:6789/0 conn(0x7fbc0c14be90 legacy=0x7fbc0c1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:14.675 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 1 monclient: found mon.c 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c143140 con 0x7fbc0c0b1730 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 10 monclient: _renew_subs 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c1d86c0 con 0x7fbc0c0b1730 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 10 monclient: _renew_subs 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c1422e0 con 0x7fbc0c0b1730 2024-03-25T19:38:14.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.674+0000 7fbc13b82640 1 librados: init done 2024-03-25T19:38:14.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.675+0000 7fbc13b82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.675+0000 7fbc13b82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.675+0000 7fbc13b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:14.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.675+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c1d7c20 con 0x7fbc0c0b1730 2024-03-25T19:38:14.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.681+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08001c70 con 0x7fbc0c0b1730 2024-03-25T19:38:14.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.681+0000 7fbc027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:14.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.681+0000 7fbc03fff640 4 set_mon_vals no callback set 2024-03-25T19:38:14.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.996+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbc08003a60 con 0x7fbc0c0b1730 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.996+0000 7fbc027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.996+0000 7fbc027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.996+0000 7fbc027fc640 10 monclient: dump: 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:14.997 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:14.998 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:14.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.997+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbc08003c40 con 0x7fbc0c0b1730 2024-03-25T19:38:14.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.997+0000 7fbc1111d640 10 monclient: get_auth_request con 0x7fbbf006df30 auth_method 0 2024-03-25T19:38:14.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:14.998+0000 7fbc027fc640 10 client.4418.objecter ms_handle_connect 0x7fbbf006df30 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fbc08054940 con 0x7fbc0c0b1730 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 10 client.4418.objecter ms_dispatch 0x7fbc0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc08050280 con 0x7fbc0c0b1730 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.006 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.005+0000 7fbc027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:15.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.245+0000 7fbc13b82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.245+0000 7fbc13b82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.245+0000 7fbc13b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:15.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.245+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc0c0627b0 con 0x7fbc0c0b1730 2024-03-25T19:38:15.278 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.277+0000 7fbc027fc640 1 -- 172.21.15.29:0/3501834601 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fbc08052380 con 0x7fbc0c0b1730 2024-03-25T19:38:15.278 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.277+0000 7fbc027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.278 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.277+0000 7fbc027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:15.278 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:15.278 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 10 librados: watch_flush enter 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 10 librados: watch_flush exit 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 10 client.4418.objecter successfully canceled tick 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbbf006df30 legacy=0x7fbbf00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 10 monclient: shutdown 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 >> v1:172.21.15.29:6790/0 conn(0x7fbc0c0b1730 legacy=0x7fbc0c13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 shutdown_connections 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 >> 172.21.15.29:0/3501834601 conn(0x7fbc0c0617a0 msgr2=0x7fbc0c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 shutdown_connections 2024-03-25T19:38:15.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 -- 172.21.15.29:0/3501834601 wait complete. 2024-03-25T19:38:15.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:15.280+0000 7fbc13b82640 1 librados: shutdown 2024-03-25T19:38:15.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:22.099 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:22.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.213+0000 7f4e0154b640 10 monclient: get_monmap_and_config 2024-03-25T19:38:22.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.213+0000 7f4e0154b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.213+0000 7f4e0154b640 10 monclient: monmap: 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:22.214853+0000 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:22.214853+0000 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:22.214 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.213+0000 7f4e0154b640 1 Processor -- start 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 1 -- start start 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient: init 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-b con 0x7f4dfc13fc60 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:22.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-c con 0x7f4dfc070570 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-a con 0x7f4dfc0727a0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc13c9f0 con 0x7f4dfc0727a0 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc06fac0 con 0x7f4dfc070570 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc0715d0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.214+0000 7f4e0154b640 10 monclient(hunting): authenticate will time out at 6378.968685s 2024-03-25T19:38:22.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.219+0000 7f4df3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4dfc070570 0x7f4dfc072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:47246/0 (socket says 172.21.15.29:47246) 2024-03-25T19:38:22.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.219+0000 7f4df3fff640 1 -- 172.21.15.29:0/1911103198 learned_addr learned my addr 172.21.15.29:0/1911103198 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:22.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393720068 0 0) 0x7f4dfc06fac0 con 0x7f4dfc070570 2024-03-25T19:38:22.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:38:22.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.231+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de8003520 con 0x7f4dfc070570 2024-03-25T19:38:22.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337619931 0 0) 0x7f4de8003520 con 0x7f4dfc070570 2024-03-25T19:38:22.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 >> v1:172.21.15.29:6790/0 conn(0x7f4dfc13fc60 legacy=0x7f4dfc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 >> v1:172.21.15.29:6789/0 conn(0x7f4dfc0727a0 legacy=0x7f4dfc14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 1 monclient: found mon.noname-c 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc0718b0 con 0x7f4dfc070570 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4e0154b640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:38:22.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.238+0000 7f4e0154b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:22.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.241+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4de4001690 con 0x7f4dfc070570 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.241+0000 7f4dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.241+0000 7f4dfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.241+0000 7f4dfa7fc640 10 monclient: dump: 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.242 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.241+0000 7f4e0154b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4de4001920 con 0x7f4dfc070570 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 1 -- 172.21.15.29:0/1911103198 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4de4001c30 con 0x7f4dfc070570 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4dfa7fc640 10 monclient: dump: 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.248 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4e0154b640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:22.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.247+0000 7f4e0154b640 4 set_mon_vals no callback set 2024-03-25T19:38:22.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 10 monclient: shutdown 2024-03-25T19:38:22.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 1 -- 172.21.15.29:0/1911103198 >> v1:172.21.15.196:6789/0 conn(0x7f4dfc070570 legacy=0x7f4dfc072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 1 -- 172.21.15.29:0/1911103198 shutdown_connections 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 1 -- 172.21.15.29:0/1911103198 >> 172.21.15.29:0/1911103198 conn(0x7f4dfc0b4a80 msgr2=0x7f4dfc0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 1 -- 172.21.15.29:0/1911103198 shutdown_connections 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.248+0000 7f4e0154b640 1 -- 172.21.15.29:0/1911103198 wait complete. 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.249+0000 7f4e0154b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.249+0000 7f4e0154b640 10 monclient: monmap: 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:22.251 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 librados: starting msgr at 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 librados: starting objecter 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 Processor -- start 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 -- start start 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 librados: setting wanted keys 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 1 librados: calling monclient init 2024-03-25T19:38:22.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 10 monclient: init 2024-03-25T19:38:22.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:22.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-a con 0x7f4dfc070570 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:22.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.250+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-c con 0x7f4dfc0727a0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:22.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 10 monclient(hunting): picked mon.noname-b con 0x7f4dfc13fc60 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:22.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc12f5a0 con 0x7f4dfc070570 2024-03-25T19:38:22.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc1340a0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc134380 con 0x7f4dfc0727a0 2024-03-25T19:38:22.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.251+0000 7f4e0154b640 10 monclient(hunting): authenticate will time out at 6379.005136s 2024-03-25T19:38:22.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.260+0000 7f4dfb7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4dfc13fc60 0x7f4dfc12ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:47284/0 (socket says 172.21.15.29:47284) 2024-03-25T19:38:22.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.260+0000 7f4dfb7fe640 1 -- 172.21.15.29:0/2398084606 learned_addr learned my addr 172.21.15.29:0/2398084606 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:22.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.262+0000 7f4df37fe640 10 client.?.objecter ms_handle_connect 0x7f4dfc13fc60 2024-03-25T19:38:22.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.262+0000 7f4df37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.271+0000 7f4df37fe640 10 client.?.objecter ms_handle_connect 0x7f4dfc070570 2024-03-25T19:38:22.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.271+0000 7f4df37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642818290 0 0) 0x7f4dfc1340a0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.277 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.277 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.277 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.277 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:38:22.277 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.276+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ddc004eb0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180666911 0 0) 0x7f4dfc12f5a0 con 0x7f4dfc070570 2024-03-25T19:38:22.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:38:22.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc1340a0 con 0x7f4dfc070570 2024-03-25T19:38:22.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 client.?.objecter ms_handle_connect 0x7f4dfc0727a0 2024-03-25T19:38:22.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.279+0000 7f4df37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375060982 0 0) 0x7f4dfc134380 con 0x7f4dfc0727a0 2024-03-25T19:38:22.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:38:22.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.287+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc12f5a0 con 0x7f4dfc0727a0 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3017778914 0 0) 0x7f4ddc004eb0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dfc134380 con 0x7f4dfc13fc60 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4df4001c70 con 0x7f4dfc13fc60 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 10 monclient(hunting): dump: 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.290 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.289+0000 7f4df37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:22.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032131123 0 0) 0x7f4dfc1340a0 con 0x7f4dfc070570 2024-03-25T19:38:22.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ddc004eb0 con 0x7f4dfc070570 2024-03-25T19:38:22.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4dec001930 con 0x7f4dfc070570 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.297+0000 7f4df37fe640 10 monclient(hunting): dump: 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.298 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804688763 0 0) 0x7f4dfc12f5a0 con 0x7f4dfc0727a0 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dfc1340a0 con 0x7f4dfc0727a0 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4de40014b0 con 0x7f4dfc0727a0 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.301+0000 7f4df37fe640 10 monclient(hunting): dump: 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.302 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.303 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.303 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.303 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.303 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.305 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46124694 0 0) 0x7f4dfc134380 con 0x7f4dfc13fc60 2024-03-25T19:38:22.305 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 >> v1:172.21.15.29:6790/0 conn(0x7f4dfc0727a0 legacy=0x7f4dfc12e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.305 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 >> v1:172.21.15.29:6789/0 conn(0x7f4dfc070570 legacy=0x7f4dfc134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.305 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.305 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 1 monclient: found mon.b 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc132c80 con 0x7f4dfc13fc60 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 10 monclient: _renew_subs 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc135200 con 0x7f4dfc13fc60 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 10 monclient: _renew_subs 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc1419e0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.304+0000 7f4e0154b640 1 librados: init done 2024-03-25T19:38:22.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.305+0000 7f4e0154b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.305+0000 7f4e0154b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.305+0000 7f4e0154b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.305+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc0627b0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df4003990 con 0x7f4dfc13fc60 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4df4003d60 con 0x7f4dfc13fc60 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df37fe640 10 monclient: dump: 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:22.579 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:22.580 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:22.580 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:22.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.578+0000 7f4df9ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:22.580 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.579+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f4df4003f80 con 0x7f4dfc13fc60 2024-03-25T19:38:22.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.579+0000 7f4dfaffd640 10 monclient: get_auth_request con 0x7f4ddc06e0f0 auth_method 0 2024-03-25T19:38:22.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.580+0000 7f4df37fe640 10 client.4423.objecter ms_handle_connect 0x7f4ddc06e0f0 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.584+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f4df4054ad0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.584+0000 7f4df37fe640 10 client.4423.objecter ms_dispatch 0x7f4dfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.584+0000 7f4df37fe640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.584+0000 7f4df37fe640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.584+0000 7f4df37fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.585+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4df40505e0 con 0x7f4dfc13fc60 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.585+0000 7f4df37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.585 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.585+0000 7f4df37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:22.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.809+0000 7f4e0154b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.809+0000 7f4e0154b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.809+0000 7f4e0154b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:22.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.809+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dfc132e10 con 0x7f4dfc13fc60 2024-03-25T19:38:22.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.832+0000 7f4df37fe640 1 -- 172.21.15.29:0/2398084606 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f4df4052510 con 0x7f4dfc13fc60 2024-03-25T19:38:22.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.832+0000 7f4df37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.832+0000 7f4df37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:22.833 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:22.833 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 10 librados: watch_flush enter 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 10 librados: watch_flush exit 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 10 client.4423.objecter successfully canceled tick 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 >> v1:172.21.15.29:6800/2314944989 conn(0x7f4ddc06e0f0 legacy=0x7f4ddc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 10 monclient: shutdown 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 >> v1:172.21.15.196:6789/0 conn(0x7f4dfc13fc60 legacy=0x7f4dfc12ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 shutdown_connections 2024-03-25T19:38:22.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 >> 172.21.15.29:0/2398084606 conn(0x7f4dfc0b4a80 msgr2=0x7f4dfc0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 shutdown_connections 2024-03-25T19:38:22.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 -- 172.21.15.29:0/2398084606 wait complete. 2024-03-25T19:38:22.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:22.834+0000 7f4e0154b640 1 librados: shutdown 2024-03-25T19:38:22.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:25.110 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:55.274 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.122 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:55.286 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.166 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:55.330 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.171 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:55.335 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.174 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:55.337 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.183 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:55.345 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.187 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:55.349 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.199 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:55.362 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:25.212 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:04:55.375 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:29.651 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:29.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.766+0000 7ffbc413d640 10 monclient: get_monmap_and_config 2024-03-25T19:38:29.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.766+0000 7ffbc413d640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.766+0000 7ffbc413d640 10 monclient: monmap: 2024-03-25T19:38:29.765 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:29.767116+0000 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:29.767116+0000 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 1 Processor -- start 2024-03-25T19:38:29.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 1 -- start start 2024-03-25T19:38:29.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 10 monclient: init 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.767+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbbc0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbbc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbbc1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc142520 con 0x7ffbbc1430c0 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0b28a0 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0b2a30 con 0x7ffbbc13f710 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.768+0000 7ffbc413d640 10 monclient(hunting): authenticate will time out at 6386.521045s 2024-03-25T19:38:29.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.772+0000 7ffbc26da640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffbbc1430c0 0x7ffbbc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48642/0 (socket says 172.21.15.29:48642) 2024-03-25T19:38:29.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.772+0000 7ffbc26da640 1 -- 172.21.15.29:0/639766433 learned_addr learned my addr 172.21.15.29:0/639766433 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021066299 0 0) 0x7ffbbc142520 con 0x7ffbbc1430c0 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:38:29.785 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.786+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba8003520 con 0x7ffbbc1430c0 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559888378 0 0) 0x7ffbbc0b28a0 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:38:29.793 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.794+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc142520 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210134272 0 0) 0x7ffba8003520 con 0x7ffbbc1430c0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 >> v1:172.21.15.29:6790/0 conn(0x7ffbbc13f710 legacy=0x7ffbbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 >> v1:172.21.15.196:6789/0 conn(0x7ffbbc0a98f0 legacy=0x7ffbbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 monclient: found mon.noname-a 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc0b44c0 con 0x7ffbbc1430c0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ffbb4001a10 con 0x7ffbbc1430c0 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc413d640 5 monclient: authenticate success, global_id 4428 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc413d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc0ed7640 10 monclient: dump: 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:29.798 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.799 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.797+0000 7ffbc413d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbb4001b70 con 0x7ffbbc1430c0 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 1 -- 172.21.15.29:0/639766433 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ffbb4001f80 con 0x7ffbbc1430c0 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 4 set_mon_vals no callback set 2024-03-25T19:38:29.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc0ed7640 10 monclient: dump: 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:29.808 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 10 monclient: shutdown 2024-03-25T19:38:29.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:29.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 1 -- 172.21.15.29:0/639766433 >> v1:172.21.15.29:6789/0 conn(0x7ffbbc1430c0 legacy=0x7ffbbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 1 -- 172.21.15.29:0/639766433 shutdown_connections 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.808+0000 7ffbc413d640 1 -- 172.21.15.29:0/639766433 >> 172.21.15.29:0/639766433 conn(0x7ffbbc0b7ea0 msgr2=0x7ffbbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.809+0000 7ffbc413d640 1 -- 172.21.15.29:0/639766433 shutdown_connections 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.809+0000 7ffbc413d640 1 -- 172.21.15.29:0/639766433 wait complete. 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.809+0000 7ffbc413d640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.809+0000 7ffbc413d640 10 monclient: monmap: 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:29.810 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 librados: starting msgr at 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 librados: starting objecter 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 Processor -- start 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 -- start start 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 librados: setting wanted keys 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 librados: calling monclient init 2024-03-25T19:38:29.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient: init 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbbc0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbbc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbbc1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0abb00 con 0x7ffbbc1430c0 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc0abde0 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc138e20 con 0x7ffbbc13f710 2024-03-25T19:38:29.812 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.813 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.810+0000 7ffbc413d640 10 monclient(hunting): authenticate will time out at 6386.563842s 2024-03-25T19:38:29.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.820+0000 7ffbc16d8640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7ffbbc13f710 0x7ffbbc136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:53466/0 (socket says 172.21.15.29:53466) 2024-03-25T19:38:29.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.820+0000 7ffbc16d8640 1 -- 172.21.15.29:0/4109559905 learned_addr learned my addr 172.21.15.29:0/4109559905 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:29.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.821+0000 7ffbb27fc640 10 client.?.objecter ms_handle_connect 0x7ffbbc13f710 2024-03-25T19:38:29.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.821+0000 7ffbb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.823+0000 7ffbb27fc640 10 client.?.objecter ms_handle_connect 0x7ffbbc0a98f0 2024-03-25T19:38:29.822 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.823+0000 7ffbb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108121747 0 0) 0x7ffbbc138e20 con 0x7ffbbc13f710 2024-03-25T19:38:29.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:38:29.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.827+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0004eb0 con 0x7ffbbc13f710 2024-03-25T19:38:29.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104825846 0 0) 0x7ffbbc0abde0 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:38:29.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.833+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc138e20 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.835+0000 7ffbb27fc640 10 client.?.objecter ms_handle_connect 0x7ffbbc1430c0 2024-03-25T19:38:29.834 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.835+0000 7ffbb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1224662087 0 0) 0x7ffba0004eb0 con 0x7ffbbc13f710 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbbc0abde0 con 0x7ffbbc13f710 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ffbac0019d0 con 0x7ffbbc13f710 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 10 monclient(hunting): dump: 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.837 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.837+0000 7ffbb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311884451 0 0) 0x7ffbbc0abb00 con 0x7ffbbc1430c0 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:38:29.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.856+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0004eb0 con 0x7ffbbc1430c0 2024-03-25T19:38:29.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175547559 0 0) 0x7ffbbc138e20 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbbc0abb00 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ffbb8001930 con 0x7ffbbc0a98f0 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.869+0000 7ffbb27fc640 10 monclient(hunting): dump: 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:29.869 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:29.870 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:29.870 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:29.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4073657640 0 0) 0x7ffbbc0abde0 con 0x7ffbbc13f710 2024-03-25T19:38:29.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 >> v1:172.21.15.196:6789/0 conn(0x7ffbbc0a98f0 legacy=0x7ffbbc1385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 >> v1:172.21.15.29:6789/0 conn(0x7ffbbc1430c0 legacy=0x7ffbbc0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 1 monclient: found mon.c 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc137b00 con 0x7ffbbc13f710 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbc413d640 5 monclient: authenticate success, global_id 4430 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbc413d640 10 monclient: _renew_subs 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbc413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.871+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc1e34b0 con 0x7ffbbc13f710 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 monclient: _renew_subs 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:29.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc1e3820 con 0x7ffbbc13f710 2024-03-25T19:38:29.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 1 librados: init done 2024-03-25T19:38:29.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:29.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:29.872+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0627b0 con 0x7ffbbc13f710 2024-03-25T19:38:30.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbac0035f0 con 0x7ffbbc13f710 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ffbac003a00 con 0x7ffbbc13f710 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb27fc640 10 monclient: dump: 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:30.176 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.176+0000 7ffbb3fff640 4 set_mon_vals no callback set 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.177+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ffbac003c20 con 0x7ffbbc13f710 2024-03-25T19:38:30.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.177+0000 7ffbc1ed9640 10 monclient: get_auth_request con 0x7ffba006dfb0 auth_method 0 2024-03-25T19:38:30.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.177+0000 7ffbb27fc640 10 client.4430.objecter ms_handle_connect 0x7ffba006dfb0 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ffbac0547d0 con 0x7ffbbc13f710 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 10 client.4430.objecter ms_dispatch 0x7ffbbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbac0502e0 con 0x7ffbbc13f710 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.183+0000 7ffbb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:30.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.434+0000 7ffbc413d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.434+0000 7ffbc413d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.434+0000 7ffbc413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:30.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.434+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbbc0b1760 con 0x7ffbbc13f710 2024-03-25T19:38:30.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.528+0000 7ffbb27fc640 1 -- 172.21.15.29:0/4109559905 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ffbac052210 con 0x7ffbbc13f710 2024-03-25T19:38:30.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.528+0000 7ffbb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.528+0000 7ffbb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:30.528 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:30.528 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:30.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 10 librados: watch_flush enter 2024-03-25T19:38:30.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 10 librados: watch_flush exit 2024-03-25T19:38:30.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 10 client.4430.objecter successfully canceled tick 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 >> v1:172.21.15.29:6800/2314944989 conn(0x7ffba006dfb0 legacy=0x7ffba0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 10 monclient: shutdown 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 >> v1:172.21.15.29:6790/0 conn(0x7ffbbc13f710 legacy=0x7ffbbc136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 shutdown_connections 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 >> 172.21.15.29:0/4109559905 conn(0x7ffbbc0b7ea0 msgr2=0x7ffbbc14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.531+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 shutdown_connections 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.532+0000 7ffbc413d640 1 -- 172.21.15.29:0/4109559905 wait complete. 2024-03-25T19:38:30.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:30.532+0000 7ffbc413d640 1 librados: shutdown 2024-03-25T19:38:30.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:37.347 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.463+0000 7f001edcd640 10 monclient: get_monmap_and_config 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.463+0000 7f001edcd640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.463+0000 7f001edcd640 10 monclient: monmap: 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:37.463 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:37.464699+0000 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:37.464699+0000 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:37.464 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.464+0000 7f001edcd640 1 Processor -- start 2024-03-25T19:38:37.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.464+0000 7f001edcd640 1 -- start start 2024-03-25T19:38:37.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient: init 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-c con 0x7f00180b1ef0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-a con 0x7f00181473a0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-b con 0x7f0018150070 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180b2570 con 0x7f00181473a0 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018146c20 con 0x7f00180b1ef0 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018146e80 con 0x7f0018150070 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.465+0000 7f001edcd640 10 monclient(hunting): authenticate will time out at 6394.218790s 2024-03-25T19:38:37.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.469+0000 7f001cb69640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f00180b1ef0 0x7f00180b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36554/0 (socket says 172.21.15.29:36554) 2024-03-25T19:38:37.469 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.469+0000 7f001cb69640 1 -- 172.21.15.29:0/2188798655 learned_addr learned my addr 172.21.15.29:0/2188798655 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944288131 0 0) 0x7f0018146c20 con 0x7f00180b1ef0 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:38:37.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.488+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008003520 con 0x7f00180b1ef0 2024-03-25T19:38:37.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846306945 0 0) 0x7f0008003520 con 0x7f00180b1ef0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 >> v1:172.21.15.29:6790/0 conn(0x7f0018150070 legacy=0x7f0018152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 >> v1:172.21.15.29:6789/0 conn(0x7f00181473a0 legacy=0x7f0018143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 monclient: found mon.noname-c 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001806e1b0 con 0x7f00180b1ef0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f000c001690 con 0x7f00180b1ef0 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:37.500 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f00177fe640 10 monclient: dump: 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f001edcd640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:38:37.501 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.500+0000 7f001edcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f000c0017f0 con 0x7f00180b1ef0 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 1 -- 172.21.15.29:0/2188798655 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f000c001c40 con 0x7f00180b1ef0 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f00177fe640 10 monclient: dump: 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.519 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f001edcd640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:37.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.519+0000 7f001edcd640 4 set_mon_vals no callback set 2024-03-25T19:38:37.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 10 monclient: shutdown 2024-03-25T19:38:37.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:37.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 1 -- 172.21.15.29:0/2188798655 >> v1:172.21.15.196:6789/0 conn(0x7f00180b1ef0 legacy=0x7f00180b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 1 -- 172.21.15.29:0/2188798655 shutdown_connections 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 1 -- 172.21.15.29:0/2188798655 >> 172.21.15.29:0/2188798655 conn(0x7f00180b7ea0 msgr2=0x7f00180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 1 -- 172.21.15.29:0/2188798655 shutdown_connections 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.520+0000 7f001edcd640 1 -- 172.21.15.29:0/2188798655 wait complete. 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.521+0000 7f001edcd640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.521+0000 7f001edcd640 10 monclient: monmap: 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:37.522 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.521+0000 7f001edcd640 1 librados: starting msgr at 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.521+0000 7f001edcd640 1 librados: starting objecter 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 Processor -- start 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 -- start start 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 librados: setting wanted keys 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 librados: calling monclient init 2024-03-25T19:38:37.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient: init 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-c con 0x7f00180b1ef0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-b con 0x7f00181473a0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient(hunting): picked mon.noname-a con 0x7f0018150070 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181dca70 con 0x7f0018150070 2024-03-25T19:38:37.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181450a0 con 0x7f00181473a0 2024-03-25T19:38:37.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181dbb60 con 0x7f00180b1ef0 2024-03-25T19:38:37.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.522+0000 7f001edcd640 10 monclient(hunting): authenticate will time out at 6394.275860s 2024-03-25T19:38:37.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.527+0000 7f001cb69640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f00180b1ef0 0x7f00181dc360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:53528/0 (socket says 172.21.15.29:53528) 2024-03-25T19:38:37.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.527+0000 7f001cb69640 1 -- 172.21.15.29:0/2953385203 learned_addr learned my addr 172.21.15.29:0/2953385203 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:37.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.529+0000 7f00157fa640 10 client.?.objecter ms_handle_connect 0x7f00180b1ef0 2024-03-25T19:38:37.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.529+0000 7f00157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.535+0000 7f00157fa640 10 client.?.objecter ms_handle_connect 0x7f00181473a0 2024-03-25T19:38:37.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.535+0000 7f00157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022609120 0 0) 0x7f00181dbb60 con 0x7f00180b1ef0 2024-03-25T19:38:37.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:38:37.539 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.539+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc004eb0 con 0x7f00180b1ef0 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457500481 0 0) 0x7f00181450a0 con 0x7f00181473a0 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00181dbb60 con 0x7f00181473a0 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 client.?.objecter ms_handle_connect 0x7f0018150070 2024-03-25T19:38:37.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.541+0000 7f00157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637662007 0 0) 0x7efffc004eb0 con 0x7f00180b1ef0 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00181450a0 con 0x7f00180b1ef0 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f000c001bc0 con 0x7f00180b1ef0 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 10 monclient(hunting): dump: 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.551 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.551+0000 7f00157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659717509 0 0) 0x7f00181dca70 con 0x7f0018150070 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:38:37.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.552+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc004eb0 con 0x7f0018150070 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2849084003 0 0) 0x7f00181450a0 con 0x7f00180b1ef0 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 >> v1:172.21.15.196:6789/0 conn(0x7f00181473a0 legacy=0x7f00180b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 >> v1:172.21.15.29:6789/0 conn(0x7f0018150070 legacy=0x7f00180b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 1 monclient: found mon.c 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00181dbe40 con 0x7f00180b1ef0 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:37.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f00157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 10 monclient: _renew_subs 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00181e7420 con 0x7f00180b1ef0 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 10 monclient: _renew_subs 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00181e7790 con 0x7f00180b1ef0 2024-03-25T19:38:37.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.557+0000 7f001edcd640 1 librados: init done 2024-03-25T19:38:37.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.558+0000 7f001edcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.558+0000 7f001edcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.558+0000 7f001edcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:37.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.558+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00180627b0 con 0x7f00180b1ef0 2024-03-25T19:38:37.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f000c004bf0 con 0x7f00180b1ef0 2024-03-25T19:38:37.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:37.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f000c004f00 con 0x7f00180b1ef0 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 10 monclient: dump: 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:37.882 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:37.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f0016ffd640 4 set_mon_vals no callback set 2024-03-25T19:38:37.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.882+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f000c01aed0 con 0x7f00180b1ef0 2024-03-25T19:38:37.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.883+0000 7f0017fff640 10 monclient: get_auth_request con 0x7efffc06de00 auth_method 0 2024-03-25T19:38:37.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.883+0000 7f00157fa640 10 client.4436.objecter ms_handle_connect 0x7efffc06de00 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.888+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f000c0549c0 con 0x7f00180b1ef0 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.888+0000 7f00157fa640 10 client.4436.objecter ms_dispatch 0x7f0018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.888+0000 7f00157fa640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.888+0000 7f00157fa640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.889+0000 7f00157fa640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T19:38:37.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.889+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f000c0504d0 con 0x7f00180b1ef0 2024-03-25T19:38:37.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.889+0000 7f00157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:37.889+0000 7f00157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:38.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.134+0000 7f001edcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.135+0000 7f001edcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.135+0000 7f001edcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.135+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00181dbfd0 con 0x7f00180b1ef0 2024-03-25T19:38:38.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.156+0000 7f00157fa640 1 -- 172.21.15.29:0/2953385203 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f000c052400 con 0x7f00180b1ef0 2024-03-25T19:38:38.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.156+0000 7f00157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.156+0000 7f00157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:38.156 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:38.156 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 10 librados: watch_flush enter 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 10 librados: watch_flush exit 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 10 client.4436.objecter successfully canceled tick 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 >> v1:172.21.15.29:6800/2314944989 conn(0x7efffc06de00 legacy=0x7efffc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 10 monclient: shutdown 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:38.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 >> v1:172.21.15.29:6790/0 conn(0x7f00180b1ef0 legacy=0x7f00181dc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 shutdown_connections 2024-03-25T19:38:38.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 >> 172.21.15.29:0/2953385203 conn(0x7f00180b7ea0 msgr2=0x7f00180b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:38.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 shutdown_connections 2024-03-25T19:38:38.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 -- 172.21.15.29:0/2953385203 wait complete. 2024-03-25T19:38:38.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:38.159+0000 7f001edcd640 1 librados: shutdown 2024-03-25T19:38:38.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:44.979 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.096+0000 7f84a888d640 10 monclient: get_monmap_and_config 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.096+0000 7f84a888d640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.096+0000 7f84a888d640 10 monclient: monmap: 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:45.097306+0000 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:45.097306+0000 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.097+0000 7f84a888d640 1 Processor -- start 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.097+0000 7f84a888d640 1 -- start start 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.097+0000 7f84a888d640 10 monclient: init 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-c con 0x7f84a40a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-a con 0x7f84a413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-b con 0x7f84a41430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a4142520 con 0x7f84a413f710 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40b28a0 con 0x7f84a40a98f0 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40b2a30 con 0x7f84a41430c0 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.098+0000 7f84a888d640 10 monclient(hunting): authenticate will time out at 6401.851418s 2024-03-25T19:38:45.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.110+0000 7f84a2d9d640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f84a41430c0 0x7f84a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:58204/0 (socket says 172.21.15.29:58204) 2024-03-25T19:38:45.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.110+0000 7f84a2d9d640 1 -- 172.21.15.29:0/2935265253 learned_addr learned my addr 172.21.15.29:0/2935265253 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:45.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781786998 0 0) 0x7f84a40b2a30 con 0x7f84a41430c0 2024-03-25T19:38:45.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.125 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:38:45.125 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.125+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c003520 con 0x7f84a41430c0 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124585180 0 0) 0x7f84a40b28a0 con 0x7f84a40a98f0 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.130+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a40b2a30 con 0x7f84a40a98f0 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668509718 0 0) 0x7f84a4142520 con 0x7f84a413f710 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:38:45.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.145+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a40b28a0 con 0x7f84a413f710 2024-03-25T19:38:45.147 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:15.310 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.205 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:15.368 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805362425 0 0) 0x7f84a40b28a0 con 0x7f84a413f710 2024-03-25T19:38:45.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 >> v1:172.21.15.29:6790/0 conn(0x7f84a41430c0 legacy=0x7f84a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 >> v1:172.21.15.196:6789/0 conn(0x7f84a40a98f0 legacy=0x7f84a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 1 monclient: found mon.noname-a 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.229+0000 7f84a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a40b44c0 con 0x7f84a413f710 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f84980015b0 con 0x7f84a413f710 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a888d640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a888d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a159a640 10 monclient: dump: 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.230 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.230+0000 7f84a888d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:45.246 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:15.410 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8498001710 con 0x7f84a413f710 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 1 -- 172.21.15.29:0/2935265253 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8498001b60 con 0x7f84a413f710 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a888d640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a888d640 4 set_mon_vals no callback set 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.247+0000 7f84a159a640 10 monclient: dump: 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.247 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:45.248 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:45.248 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:45.248 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 10 monclient: shutdown 2024-03-25T19:38:45.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 1 -- 172.21.15.29:0/2935265253 >> v1:172.21.15.29:6789/0 conn(0x7f84a413f710 legacy=0x7f84a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 1 -- 172.21.15.29:0/2935265253 shutdown_connections 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 1 -- 172.21.15.29:0/2935265253 >> 172.21.15.29:0/2935265253 conn(0x7f84a40b7ea0 msgr2=0x7f84a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 1 -- 172.21.15.29:0/2935265253 shutdown_connections 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.248+0000 7f84a888d640 1 -- 172.21.15.29:0/2935265253 wait complete. 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient: monmap: 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:45.250 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 librados: starting msgr at 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 librados: starting objecter 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 Processor -- start 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 -- start start 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 librados: setting wanted keys 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 librados: calling monclient init 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient: init 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-c con 0x7f84a40a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-b con 0x7f84a413f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:45.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 10 monclient(hunting): picked mon.noname-a con 0x7f84a41430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:45.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a41d8340 con 0x7f84a41430c0 2024-03-25T19:38:45.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a41d7710 con 0x7f84a413f710 2024-03-25T19:38:45.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.249+0000 7f84a888d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a4138e90 con 0x7f84a40a98f0 2024-03-25T19:38:45.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.250+0000 7f84a888d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.250+0000 7f84a888d640 10 monclient(hunting): authenticate will time out at 6402.002876s 2024-03-25T19:38:45.257 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:15.419 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.273 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:15.436 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.276 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:15.440 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.284+0000 7f84a1d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f84a413f710 0x7f84a4138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41460/0 (socket says 172.21.15.29:41460) 2024-03-25T19:38:45.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.284+0000 7f84a1d9b640 1 -- 172.21.15.29:0/1952690083 learned_addr learned my addr 172.21.15.29:0/1952690083 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:45.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.286+0000 7f8492ffd640 10 client.?.objecter ms_handle_connect 0x7f84a413f710 2024-03-25T19:38:45.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.286+0000 7f8492ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.286 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:15.450 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.298 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:15.461 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.300+0000 7f8492ffd640 10 client.?.objecter ms_handle_connect 0x7f84a41430c0 2024-03-25T19:38:45.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.300+0000 7f8492ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.321+0000 7f8492ffd640 10 client.?.objecter ms_handle_connect 0x7f84a40a98f0 2024-03-25T19:38:45.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.321+0000 7f8492ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.320 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:15.484 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940921839 0 0) 0x7f84a41d7710 con 0x7f84a413f710 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:38:45.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.333+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484004eb0 con 0x7f84a413f710 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196203205 0 0) 0x7f84a41d8340 con 0x7f84a41430c0 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.343+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a41d7710 con 0x7f84a41430c0 2024-03-25T19:38:45.343 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591406442 0 0) 0x7f84a4138e90 con 0x7f84a40a98f0 2024-03-25T19:38:45.344 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.344 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.344 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.344 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:38:45.344 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.344+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a41d8340 con 0x7f84a40a98f0 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520547618 0 0) 0x7f8484004eb0 con 0x7f84a413f710 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a4138e90 con 0x7f84a413f710 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8498001b20 con 0x7f84a413f710 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:45.389 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:45.390 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:45.390 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.389+0000 7f8492ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 186034370 0 0) 0x7f84a41d7710 con 0x7f84a41430c0 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8484004eb0 con 0x7f84a41430c0 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f849c001ab0 con 0x7f84a41430c0 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.401+0000 7f8492ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.401 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:45.402 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.422+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 645356790 0 0) 0x7f84a4138e90 con 0x7f84a413f710 2024-03-25T19:38:45.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.422+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 >> v1:172.21.15.29:6790/0 conn(0x7f84a40a98f0 legacy=0x7f84a41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.422+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 >> v1:172.21.15.29:6789/0 conn(0x7f84a41430c0 legacy=0x7f84a4133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.422+0000 7f8492ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.422+0000 7f8492ffd640 1 monclient: found mon.b 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4139020 con 0x7f84a413f710 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f8492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 10 monclient: _renew_subs 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4137f30 con 0x7f84a413f710 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 10 monclient: _renew_subs 2024-03-25T19:38:45.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a41348e0 con 0x7f84a413f710 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.423+0000 7f84a888d640 1 librados: init done 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.424+0000 7f84a888d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.424+0000 7f84a888d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.424+0000 7f84a888d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:45.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.424+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a414f2d0 con 0x7f84a413f710 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84980038a0 con 0x7f84a413f710 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f8498003c30 con 0x7f84a413f710 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8492ffd640 10 monclient: dump: 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.716+0000 7f8493fff640 4 set_mon_vals no callback set 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.717+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f8498003e50 con 0x7f84a413f710 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.718+0000 7f84a259c640 10 monclient: get_auth_request con 0x7f848406df90 auth_method 0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.718+0000 7f8492ffd640 10 client.4441.objecter ms_handle_connect 0x7f848406df90 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.723+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f84980548d0 con 0x7f84a413f710 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.723+0000 7f8492ffd640 10 client.4441.objecter ms_dispatch 0x7f84a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.723+0000 7f8492ffd640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.723+0000 7f8492ffd640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.723+0000 7f8492ffd640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T19:38:45.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.724+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84980503e0 con 0x7f84a413f710 2024-03-25T19:38:45.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.724+0000 7f8492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.724+0000 7f8492ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:45.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.972+0000 7f84a888d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.972+0000 7f84a888d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.972+0000 7f84a888d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:38:45.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.972+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84a40627b0 con 0x7f84a413f710 2024-03-25T19:38:45.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.993+0000 7f8492ffd640 1 -- 172.21.15.29:0/1952690083 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f8498052310 con 0x7f84a413f710 2024-03-25T19:38:45.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.993+0000 7f8492ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.993+0000 7f8492ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:45.993 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:45.993 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:45.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.995+0000 7f84a888d640 10 librados: watch_flush enter 2024-03-25T19:38:45.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.995+0000 7f84a888d640 10 librados: watch_flush exit 2024-03-25T19:38:45.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.995+0000 7f84a888d640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.995+0000 7f84a888d640 10 client.4441.objecter successfully canceled tick 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 >> v1:172.21.15.29:6800/2314944989 conn(0x7f848406df90 legacy=0x7f8484070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 10 monclient: shutdown 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 >> v1:172.21.15.196:6789/0 conn(0x7f84a413f710 legacy=0x7f84a4138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 shutdown_connections 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 >> 172.21.15.29:0/1952690083 conn(0x7f84a40b7ea0 msgr2=0x7f84a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 shutdown_connections 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 -- 172.21.15.29:0/1952690083 wait complete. 2024-03-25T19:38:45.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:45.996+0000 7f84a888d640 1 librados: shutdown 2024-03-25T19:38:46.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:52.813 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.911+0000 7facd1dca640 10 monclient: get_monmap_and_config 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.911+0000 7facd1dca640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.911+0000 7facd1dca640 10 monclient: monmap: 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:38:52.912628+0000 2024-03-25T19:38:52.911 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:38:52.912628+0000 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.912+0000 7facd1dca640 1 Processor -- start 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.912+0000 7facd1dca640 1 -- start start 2024-03-25T19:38:52.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.912+0000 7facd1dca640 10 monclient: init 2024-03-25T19:38:52.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:52.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-a con 0x7faccc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-b con 0x7faccc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-c con 0x7faccc1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc142520 con 0x7faccc0a98f0 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b28a0 con 0x7faccc1430c0 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b2a30 con 0x7faccc13f710 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.913+0000 7facd1dca640 10 monclient(hunting): authenticate will time out at 6409.666765s 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.915+0000 7faccbfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7faccc1430c0 0x7faccc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:55118/0 (socket says 172.21.15.29:55118) 2024-03-25T19:38:52.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.915+0000 7faccbfff640 1 -- 172.21.15.29:0/3426088133 learned_addr learned my addr 172.21.15.29:0/3426088133 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082031657 0 0) 0x7faccc0b28a0 con 0x7faccc1430c0 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:38:52.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.924+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4003520 con 0x7faccc1430c0 2024-03-25T19:38:52.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789166807 0 0) 0x7faccc0b2a30 con 0x7faccc13f710 2024-03-25T19:38:52.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:38:52.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.933+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc0b28a0 con 0x7faccc13f710 2024-03-25T19:38:52.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626859093 0 0) 0x7facb4003520 con 0x7faccc1430c0 2024-03-25T19:38:52.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 >> v1:172.21.15.29:6790/0 conn(0x7faccc13f710 legacy=0x7faccc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 >> v1:172.21.15.29:6789/0 conn(0x7faccc0a98f0 legacy=0x7faccc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0b44c0 con 0x7faccc1430c0 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7facc4001a10 con 0x7faccc1430c0 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.935+0000 7facca7fc640 10 monclient: dump: 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.936 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.936+0000 7facd1dca640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:38:52.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.936+0000 7facd1dca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc4001b70 con 0x7faccc1430c0 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 1 -- 172.21.15.29:0/3426088133 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7facc4001f80 con 0x7faccc1430c0 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facd1dca640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facca7fc640 10 monclient: dump: 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:52.941 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:52.942 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:52.942 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:52.942 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:52.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.941+0000 7facd1dca640 4 set_mon_vals no callback set 2024-03-25T19:38:52.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 10 monclient: shutdown 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 1 -- 172.21.15.29:0/3426088133 >> v1:172.21.15.196:6789/0 conn(0x7faccc1430c0 legacy=0x7faccc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 1 -- 172.21.15.29:0/3426088133 shutdown_connections 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 1 -- 172.21.15.29:0/3426088133 >> 172.21.15.29:0/3426088133 conn(0x7faccc0b7ea0 msgr2=0x7faccc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 1 -- 172.21.15.29:0/3426088133 shutdown_connections 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.942+0000 7facd1dca640 1 -- 172.21.15.29:0/3426088133 wait complete. 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.943+0000 7facd1dca640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.943+0000 7facd1dca640 10 monclient: monmap: 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:38:52.943 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:38:52.944 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:38:52.944 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.943+0000 7facd1dca640 1 librados: starting msgr at 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.943+0000 7facd1dca640 1 librados: starting objecter 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.944+0000 7facd1dca640 1 Processor -- start 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.944+0000 7facd1dca640 1 -- start start 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.944+0000 7facd1dca640 1 librados: setting wanted keys 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.944+0000 7facd1dca640 1 librados: calling monclient init 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.944+0000 7facd1dca640 10 monclient: init 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-a con 0x7faccc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-c con 0x7faccc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient(hunting): picked mon.noname-b con 0x7faccc1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc1369d0 con 0x7faccc0a98f0 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc133a60 con 0x7faccc1430c0 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0ab9e0 con 0x7faccc13f710 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.945+0000 7facd1dca640 10 monclient(hunting): authenticate will time out at 6409.698353s 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.948+0000 7faccaffd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7faccc13f710 0x7faccc1362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:47080/0 (socket says 172.21.15.29:47080) 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.948+0000 7faccaffd640 1 -- 172.21.15.29:0/1200877749 learned_addr learned my addr 172.21.15.29:0/1200877749 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:38:52.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.948+0000 7facb3fff640 10 client.?.objecter ms_handle_connect 0x7faccc13f710 2024-03-25T19:38:52.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.948+0000 7facb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.953+0000 7facb3fff640 10 client.?.objecter ms_handle_connect 0x7faccc0a98f0 2024-03-25T19:38:52.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.953+0000 7facb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699257227 0 0) 0x7faccc0ab9e0 con 0x7faccc13f710 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:38:52.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.957+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7faccc13f710 2024-03-25T19:38:52.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.961+0000 7facb3fff640 10 client.?.objecter ms_handle_connect 0x7faccc1430c0 2024-03-25T19:38:52.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.961+0000 7facb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 79411479 0 0) 0x7facb8004eb0 con 0x7faccc13f710 2024-03-25T19:38:52.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc0ab9e0 con 0x7faccc13f710 2024-03-25T19:38:52.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7facc0001a10 con 0x7faccc13f710 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055024523 0 0) 0x7faccc1369d0 con 0x7faccc0a98f0 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:38:52.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.964+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7faccc0a98f0 2024-03-25T19:38:52.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 43301827 0 0) 0x7faccc0ab9e0 con 0x7faccc13f710 2024-03-25T19:38:52.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 >> v1:172.21.15.196:6789/0 conn(0x7faccc1430c0 legacy=0x7faccc133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 >> v1:172.21.15.29:6789/0 conn(0x7faccc0a98f0 legacy=0x7faccc132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 1 monclient: found mon.c 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0abcc0 con 0x7faccc13f710 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facd1dca640 5 monclient: authenticate success, global_id 4448 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facd1dca640 10 monclient: _renew_subs 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facd1dca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.969+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc0abfa0 con 0x7faccc13f710 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 10 monclient: _renew_subs 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:52.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc137e40 con 0x7faccc13f710 2024-03-25T19:38:52.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 1 librados: init done 2024-03-25T19:38:52.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.970+0000 7facd1dca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.971+0000 7facd1dca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.971+0000 7facd1dca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:52.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:52.971+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc0627b0 con 0x7faccc13f710 2024-03-25T19:38:53.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc0001c90 con 0x7faccc13f710 2024-03-25T19:38:53.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:53.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7facc0003a60 con 0x7faccc13f710 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facb3fff640 10 monclient: dump: 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:38:53.301 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:38:53.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.300+0000 7facc9ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:53.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.301+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7facc0003c80 con 0x7faccc13f710 2024-03-25T19:38:53.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.302+0000 7faccb7fe640 10 monclient: get_auth_request con 0x7facb806dd90 auth_method 0 2024-03-25T19:38:53.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.302+0000 7facb3fff640 10 client.4448.objecter ms_handle_connect 0x7facb806dd90 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7facc0054860 con 0x7faccc13f710 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 10 client.4448.objecter ms_dispatch 0x7faccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc0050370 con 0x7faccc13f710 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.308+0000 7facb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:53.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.574+0000 7facd1dca640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.574+0000 7facd1dca640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.574+0000 7facd1dca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:38:53.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.574+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faccc13dfd0 con 0x7faccc13f710 2024-03-25T19:38:53.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.602+0000 7facb3fff640 1 -- 172.21.15.29:0/1200877749 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7facc00522a0 con 0x7faccc13f710 2024-03-25T19:38:53.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.602+0000 7facb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.602+0000 7facb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:53.602 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:38:53.602 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 10 librados: watch_flush enter 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 10 librados: watch_flush exit 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 10 client.4448.objecter successfully canceled tick 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 >> v1:172.21.15.29:6800/2314944989 conn(0x7facb806dd90 legacy=0x7facb8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 10 monclient: shutdown 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 >> v1:172.21.15.29:6790/0 conn(0x7faccc13f710 legacy=0x7faccc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 shutdown_connections 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.604+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 >> 172.21.15.29:0/1200877749 conn(0x7faccc0b7ea0 msgr2=0x7faccc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.605+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 shutdown_connections 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.605+0000 7facd1dca640 1 -- 172.21.15.29:0/1200877749 wait complete. 2024-03-25T19:38:53.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:38:53.605+0000 7facd1dca640 1 librados: shutdown 2024-03-25T19:38:53.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:00.422 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:00.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.521+0000 7ff200686640 10 monclient: get_monmap_and_config 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.521+0000 7ff200686640 10 monclient: build_initial_monmap 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.521+0000 7ff200686640 10 monclient: monmap: 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:00.522204+0000 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:00.522204+0000 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:00.521 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.521+0000 7ff200686640 1 Processor -- start 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 1 -- start start 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient: init 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f80a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f81430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f8142520 con 0x7ff1f80a98f0 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80b28a0 con 0x7ff1f813f710 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80b2a30 con 0x7ff1f81430c0 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:00.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.522+0000 7ff200686640 10 monclient(hunting): authenticate will time out at 6417.275661s 2024-03-25T19:39:00.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.523+0000 7ff1fec23640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7ff1f81430c0 0x7ff1f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:36072/0 (socket says 172.21.15.29:36072) 2024-03-25T19:39:00.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.523+0000 7ff1fec23640 1 -- 172.21.15.29:0/4228681345 learned_addr learned my addr 172.21.15.29:0/4228681345 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303788272 0 0) 0x7ff1f80b28a0 con 0x7ff1f813f710 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:39:00.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.541+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4003520 con 0x7ff1f813f710 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685145098 0 0) 0x7ff1f80b2a30 con 0x7ff1f81430c0 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:39:00.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.545+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f80b28a0 con 0x7ff1f81430c0 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939867370 0 0) 0x7ff1f8142520 con 0x7ff1f80a98f0 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:39:00.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.558+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f80b2a30 con 0x7ff1f80a98f0 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584287017 0 0) 0x7ff1f80b28a0 con 0x7ff1f81430c0 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 >> v1:172.21.15.196:6789/0 conn(0x7ff1f813f710 legacy=0x7ff1f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 >> v1:172.21.15.29:6789/0 conn(0x7ff1f80a98f0 legacy=0x7ff1f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: _finish_hunting 0 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 monclient: found mon.noname-b 2024-03-25T19:39:00.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80b44c0 con 0x7ff1f81430c0 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: _finish_auth 0 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: _check_auth_tickets 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f00019f0 con 0x7ff1f81430c0 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff1fd420640 10 monclient: dump: 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.560 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff200686640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:39:00.561 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.559+0000 7ff200686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:00.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f0001b50 con 0x7ff1f81430c0 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 1 -- 172.21.15.29:0/4228681345 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f0001f60 con 0x7ff1f81430c0 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff200686640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff200686640 4 set_mon_vals no callback set 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.568+0000 7ff1fd420640 10 monclient: dump: 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.568 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.569 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.569 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 10 monclient: shutdown 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 1 -- 172.21.15.29:0/4228681345 >> v1:172.21.15.29:6790/0 conn(0x7ff1f81430c0 legacy=0x7ff1f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 1 -- 172.21.15.29:0/4228681345 shutdown_connections 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 1 -- 172.21.15.29:0/4228681345 >> 172.21.15.29:0/4228681345 conn(0x7ff1f80b7ea0 msgr2=0x7ff1f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 1 -- 172.21.15.29:0/4228681345 shutdown_connections 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.569+0000 7ff200686640 1 -- 172.21.15.29:0/4228681345 wait complete. 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 10 monclient: build_initial_monmap 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 10 monclient: monmap: 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:00.571 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 librados: starting msgr at 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 librados: starting objecter 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 Processor -- start 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 -- start start 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 librados: setting wanted keys 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 1 librados: calling monclient init 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.570+0000 7ff200686640 10 monclient: init 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:00.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f80a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f81430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f8138600 con 0x7ff1f81430c0 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80b1620 con 0x7ff1f813f710 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f81346b0 con 0x7ff1f80a98f0 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:00.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.571+0000 7ff200686640 10 monclient(hunting): authenticate will time out at 6417.324296s 2024-03-25T19:39:00.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.579+0000 7ff1fec23640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff1f81430c0 0x7ff1f81da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46468/0 (socket says 172.21.15.29:46468) 2024-03-25T19:39:00.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.579+0000 7ff1fec23640 1 -- 172.21.15.29:0/3993491101 learned_addr learned my addr 172.21.15.29:0/3993491101 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:00.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.580+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f81430c0 2024-03-25T19:39:00.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.580+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f813f710 2024-03-25T19:39:00.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320937411 0 0) 0x7ff1f8138600 con 0x7ff1f81430c0 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.588+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:39:00.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.589+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1f81430c0 2024-03-25T19:39:00.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.590+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f80a98f0 2024-03-25T19:39:00.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.590+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965601118 0 0) 0x7ff1f80b1620 con 0x7ff1f813f710 2024-03-25T19:39:00.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:39:00.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.604+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8138600 con 0x7ff1f813f710 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613696933 0 0) 0x7ff1f81346b0 con 0x7ff1f80a98f0 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:39:00.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.606+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f80b1620 con 0x7ff1f80a98f0 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464711795 0 0) 0x7ff1dc004eb0 con 0x7ff1f81430c0 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f81346b0 con 0x7ff1f81430c0 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f0001f60 con 0x7ff1f81430c0 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.610 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.610+0000 7ff1eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:00.618 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 148564412 0 0) 0x7ff1f8138600 con 0x7ff1f813f710 2024-03-25T19:39:00.618 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1f813f710 2024-03-25T19:39:00.618 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f4001950 con 0x7ff1f813f710 2024-03-25T19:39:00.618 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.618 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.618+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.619 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438332154 0 0) 0x7ff1f80b1620 con 0x7ff1f80a98f0 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f8138600 con 0x7ff1f80a98f0 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1e8001830 con 0x7ff1f80a98f0 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.619+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.620 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.621 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504641383 0 0) 0x7ff1f81346b0 con 0x7ff1f81430c0 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 >> v1:172.21.15.29:6790/0 conn(0x7ff1f80a98f0 legacy=0x7ff1f8137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 >> v1:172.21.15.196:6789/0 conn(0x7ff1f813f710 legacy=0x7ff1f8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 1 monclient: found mon.a 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8134840 con 0x7ff1f81430c0 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:00.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff1eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 10 monclient: _renew_subs 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f8137f50 con 0x7ff1f81430c0 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 10 monclient: _renew_subs 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f81daf90 con 0x7ff1f81430c0 2024-03-25T19:39:00.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.623+0000 7ff200686640 1 librados: init done 2024-03-25T19:39:00.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.624+0000 7ff200686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.624+0000 7ff200686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.624+0000 7ff200686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:00.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.624+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f80627b0 con 0x7ff1f81430c0 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f0004e10 con 0x7ff1f81430c0 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f0005120 con 0x7ff1f81430c0 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1eeffd640 10 monclient: dump: 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:00.953 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.953+0000 7ff1fcc1f640 4 set_mon_vals no callback set 2024-03-25T19:39:00.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.954+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ff1f001b0f0 con 0x7ff1f81430c0 2024-03-25T19:39:00.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.954+0000 7ff1fe422640 10 monclient: get_auth_request con 0x7ff1dc06e0f0 auth_method 0 2024-03-25T19:39:00.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.955+0000 7ff1eeffd640 10 client.4455.objecter ms_handle_connect 0x7ff1dc06e0f0 2024-03-25T19:39:00.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ff1f0054c00 con 0x7ff1f81430c0 2024-03-25T19:39:00.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 10 client.4455.objecter ms_dispatch 0x7ff1f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1f0050710 con 0x7ff1f81430c0 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:00.961+0000 7ff1eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:01.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.215+0000 7ff200686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.215+0000 7ff200686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.215+0000 7ff200686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:01.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.215+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1f814f300 con 0x7ff1f81430c0 2024-03-25T19:39:01.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.238+0000 7ff1eeffd640 1 -- 172.21.15.29:0/3993491101 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ff1f0052640 con 0x7ff1f81430c0 2024-03-25T19:39:01.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.238+0000 7ff1eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.238+0000 7ff1eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:01.238 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:01.238 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:01.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 10 librados: watch_flush enter 2024-03-25T19:39:01.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 10 librados: watch_flush exit 2024-03-25T19:39:01.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 >> v1:172.21.15.29:6800/2314944989 conn(0x7ff1dc06e0f0 legacy=0x7ff1dc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 10 monclient: shutdown 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.240+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 >> v1:172.21.15.29:6789/0 conn(0x7ff1f81430c0 legacy=0x7ff1f81da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.241+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 shutdown_connections 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.241+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 >> 172.21.15.29:0/3993491101 conn(0x7ff1f80b7ea0 msgr2=0x7ff1f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.241+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 shutdown_connections 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.241+0000 7ff200686640 1 -- 172.21.15.29:0/3993491101 wait complete. 2024-03-25T19:39:01.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:01.241+0000 7ff200686640 1 librados: shutdown 2024-03-25T19:39:01.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:05.225 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:35.389 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.269 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:35.433 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.296 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:35.460 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.331 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:35.493 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.336 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:35.499 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.352 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:35.516 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.365 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:35.529 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.369 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:05:35.531 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:05.396 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:35.560 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:08.057 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.173+0000 7f918abe8640 10 monclient: get_monmap_and_config 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.173+0000 7f918abe8640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.173+0000 7f918abe8640 10 monclient: monmap: 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:08.175540+0000 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:08.175540+0000 2024-03-25T19:39:08.174 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.175 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:08.175 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:08.175 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:08.175 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:08.175 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.174+0000 7f918abe8640 1 Processor -- start 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.174+0000 7f918abe8640 1 -- start start 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.174+0000 7f918abe8640 10 monclient: init 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-c con 0x7f91840a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:08.176 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-a con 0x7f918413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-b con 0x7f91841430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9184142520 con 0x7f918413f710 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91840b28a0 con 0x7f91840a98f0 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91840b2a30 con 0x7f91841430c0 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.175+0000 7f918abe8640 10 monclient(hunting): authenticate will time out at 6424.929448s 2024-03-25T19:39:08.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.186+0000 7f9188984640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f91840a98f0 0x7f91840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41860/0 (socket says 172.21.15.29:41860) 2024-03-25T19:39:08.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.186+0000 7f9188984640 1 -- 172.21.15.29:0/912733417 learned_addr learned my addr 172.21.15.29:0/912733417 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247330252 0 0) 0x7f91840b28a0 con 0x7f91840a98f0 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:39:08.194 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.193+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c003520 con 0x7f91840a98f0 2024-03-25T19:39:08.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877463049 0 0) 0x7f9184142520 con 0x7f918413f710 2024-03-25T19:39:08.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:39:08.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.206+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91840b28a0 con 0x7f918413f710 2024-03-25T19:39:08.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885779916 0 0) 0x7f916c003520 con 0x7f91840a98f0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 >> v1:172.21.15.29:6790/0 conn(0x7f91841430c0 legacy=0x7f918414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 >> v1:172.21.15.29:6789/0 conn(0x7f918413f710 legacy=0x7f91841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840b44c0 con 0x7f91840a98f0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9180001690 con 0x7f91840a98f0 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f917b7fe640 10 monclient: dump: 2024-03-25T19:39:08.209 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f918abe8640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:39:08.210 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.208+0000 7f918abe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91800017f0 con 0x7f91840a98f0 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 1 -- 172.21.15.29:0/912733417 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9180001c00 con 0x7f91840a98f0 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f917b7fe640 10 monclient: dump: 2024-03-25T19:39:08.226 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f918abe8640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:08.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.225+0000 7f918abe8640 4 set_mon_vals no callback set 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 10 monclient: shutdown 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 1 -- 172.21.15.29:0/912733417 >> v1:172.21.15.196:6789/0 conn(0x7f91840a98f0 legacy=0x7f91840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 1 -- 172.21.15.29:0/912733417 shutdown_connections 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 1 -- 172.21.15.29:0/912733417 >> 172.21.15.29:0/912733417 conn(0x7f91840b7ea0 msgr2=0x7f91840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 1 -- 172.21.15.29:0/912733417 shutdown_connections 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.226+0000 7f918abe8640 1 -- 172.21.15.29:0/912733417 wait complete. 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.227+0000 7f918abe8640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.227+0000 7f918abe8640 10 monclient: monmap: 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:08.229 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 librados: starting msgr at 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 librados: starting objecter 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 Processor -- start 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 -- start start 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 librados: setting wanted keys 2024-03-25T19:39:08.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 1 librados: calling monclient init 2024-03-25T19:39:08.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.228+0000 7f918abe8640 10 monclient: init 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-b con 0x7f91840a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-a con 0x7f918413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient(hunting): picked mon.noname-c con 0x7f91841430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9184139610 con 0x7f918413f710 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9184139900 con 0x7f91840a98f0 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9184139a90 con 0x7f91841430c0 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.229+0000 7f918abe8640 10 monclient(hunting): authenticate will time out at 6424.983573s 2024-03-25T19:39:08.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.238+0000 7f917bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f918413f710 0x7f9184132800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40762/0 (socket says 172.21.15.29:40762) 2024-03-25T19:39:08.238 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.238+0000 7f917bfff640 1 -- 172.21.15.29:0/256083083 learned_addr learned my addr 172.21.15.29:0/256083083 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:08.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.241+0000 7f91797fa640 10 client.?.objecter ms_handle_connect 0x7f918413f710 2024-03-25T19:39:08.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.241+0000 7f91797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.247+0000 7f91797fa640 10 client.?.objecter ms_handle_connect 0x7f91841430c0 2024-03-25T19:39:08.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.247+0000 7f91797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.248+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218826148 0 0) 0x7f9184139610 con 0x7f918413f710 2024-03-25T19:39:08.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.248+0000 7f91797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.248+0000 7f91797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.248+0000 7f91797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.248+0000 7f91797fa640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:39:08.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.249+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9170004eb0 con 0x7f918413f710 2024-03-25T19:39:08.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.253+0000 7f91797fa640 10 client.?.objecter ms_handle_connect 0x7f91840a98f0 2024-03-25T19:39:08.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.253+0000 7f91797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3074693632 0 0) 0x7f9170004eb0 con 0x7f918413f710 2024-03-25T19:39:08.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9184139610 con 0x7f918413f710 2024-03-25T19:39:08.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9174001a10 con 0x7f918413f710 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 10 monclient(hunting): dump: 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.257 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.256+0000 7f91797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3581435672 0 0) 0x7f9184139610 con 0x7f918413f710 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 >> v1:172.21.15.29:6790/0 conn(0x7f91841430c0 legacy=0x7f9184133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 >> v1:172.21.15.196:6789/0 conn(0x7f91840a98f0 legacy=0x7f91841320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 1 monclient: found mon.a 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184138bf0 con 0x7f918413f710 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f91797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 10 monclient: _renew_subs 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9184138e40 con 0x7f918413f710 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 10 monclient: _renew_subs 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9184138380 con 0x7f918413f710 2024-03-25T19:39:08.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.267+0000 7f918abe8640 1 librados: init done 2024-03-25T19:39:08.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.268+0000 7f918abe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.268+0000 7f918abe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.268+0000 7f918abe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:08.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.268+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91840627b0 con 0x7f918413f710 2024-03-25T19:39:08.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9174001c90 con 0x7f918413f710 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9174003a60 con 0x7f918413f710 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f91797fa640 10 monclient: dump: 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:08.595 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:08.596 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:08.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.594+0000 7f917affd640 4 set_mon_vals no callback set 2024-03-25T19:39:08.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.595+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9174003c80 con 0x7f918413f710 2024-03-25T19:39:08.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.595+0000 7f9188984640 10 monclient: get_auth_request con 0x7f917006dd00 auth_method 0 2024-03-25T19:39:08.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.596+0000 7f91797fa640 10 client.4461.objecter ms_handle_connect 0x7f917006dd00 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f9174054840 con 0x7f918413f710 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 10 client.4461.objecter ms_dispatch 0x7f9184000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9174050350 con 0x7f918413f710 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.601+0000 7f91797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:08.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.856+0000 7f918abe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.856+0000 7f918abe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.856+0000 7f918abe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:08.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.856+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918413df10 con 0x7f918413f710 2024-03-25T19:39:08.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.887+0000 7f91797fa640 1 -- 172.21.15.29:0/256083083 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f9174052280 con 0x7f918413f710 2024-03-25T19:39:08.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.887+0000 7f91797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.887+0000 7f91797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:08.888 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:08.888 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:08.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 10 librados: watch_flush enter 2024-03-25T19:39:08.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 10 librados: watch_flush exit 2024-03-25T19:39:08.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T19:39:08.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 10 client.4461.objecter successfully canceled tick 2024-03-25T19:39:08.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 >> v1:172.21.15.29:6800/2314944989 conn(0x7f917006dd00 legacy=0x7f91700701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 10 monclient: shutdown 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 >> v1:172.21.15.29:6789/0 conn(0x7f918413f710 legacy=0x7f9184132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 shutdown_connections 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.890+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 >> 172.21.15.29:0/256083083 conn(0x7f91840b7ea0 msgr2=0x7f9184074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.891+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 shutdown_connections 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.891+0000 7f918abe8640 1 -- 172.21.15.29:0/256083083 wait complete. 2024-03-25T19:39:08.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:08.891+0000 7f918abe8640 1 librados: shutdown 2024-03-25T19:39:08.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:15.710 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.828+0000 7f5053093640 10 monclient: get_monmap_and_config 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.828+0000 7f5053093640 10 monclient: build_initial_monmap 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.828+0000 7f5053093640 10 monclient: monmap: 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:15.830458+0000 2024-03-25T19:39:15.829 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:15.830458+0000 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:15.830 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.829+0000 7f5053093640 1 Processor -- start 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.829+0000 7f5053093640 1 -- start start 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.829+0000 7f5053093640 10 monclient: init 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient(hunting): picked mon.noname-a con 0x7f504c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:15.831 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient(hunting): picked mon.noname-b con 0x7f504c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient(hunting): picked mon.noname-c con 0x7f504c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c142520 con 0x7f504c0a98f0 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c0b28a0 con 0x7f504c1430c0 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c0b2a30 con 0x7f504c13f710 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:15.832 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.830+0000 7f5053093640 10 monclient(hunting): authenticate will time out at 6432.584472s 2024-03-25T19:39:15.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.837+0000 7f5050e2f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f504c0a98f0 0x7f504c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45448/0 (socket says 172.21.15.29:45448) 2024-03-25T19:39:15.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.837+0000 7f5050e2f640 1 -- 172.21.15.29:0/1715300946 learned_addr learned my addr 172.21.15.29:0/1715300946 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65217941 0 0) 0x7f504c0b28a0 con 0x7f504c1430c0 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:39:15.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.854+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c003520 con 0x7f504c1430c0 2024-03-25T19:39:15.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696899781 0 0) 0x7f504c142520 con 0x7f504c0a98f0 2024-03-25T19:39:15.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:39:15.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.861+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c0b28a0 con 0x7f504c0a98f0 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738604711 0 0) 0x7f504c0b2a30 con 0x7f504c13f710 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:39:15.866 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.865+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c142520 con 0x7f504c13f710 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057850174 0 0) 0x7f503c003520 con 0x7f504c1430c0 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 >> v1:172.21.15.29:6790/0 conn(0x7f504c13f710 legacy=0x7f504c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 >> v1:172.21.15.29:6789/0 conn(0x7f504c0a98f0 legacy=0x7f504c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:15.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c0b44c0 con 0x7f504c1430c0 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.871+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5044001a10 con 0x7f504c1430c0 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.872+0000 7f504b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.872+0000 7f504b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.872+0000 7f504b7fe640 10 monclient: dump: 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:15.873 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:15.874 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:15.874 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.874 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.872+0000 7f5053093640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:39:15.874 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.872+0000 7f5053093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5044001b70 con 0x7f504c1430c0 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 1 -- 172.21.15.29:0/1715300946 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5044001f80 con 0x7f504c1430c0 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f504b7fe640 10 monclient: dump: 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:15.881 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:15.882 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f5053093640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:15.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.880+0000 7f5053093640 4 set_mon_vals no callback set 2024-03-25T19:39:15.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 10 monclient: shutdown 2024-03-25T19:39:15.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:15.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 1 -- 172.21.15.29:0/1715300946 >> v1:172.21.15.196:6789/0 conn(0x7f504c1430c0 legacy=0x7f504c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 1 -- 172.21.15.29:0/1715300946 shutdown_connections 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 1 -- 172.21.15.29:0/1715300946 >> 172.21.15.29:0/1715300946 conn(0x7f504c0b7ea0 msgr2=0x7f504c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 1 -- 172.21.15.29:0/1715300946 shutdown_connections 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.881+0000 7f5053093640 1 -- 172.21.15.29:0/1715300946 wait complete. 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 10 monclient: build_initial_monmap 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 10 monclient: monmap: 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:15.884 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 librados: starting msgr at 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 librados: starting objecter 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 Processor -- start 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 -- start start 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 librados: setting wanted keys 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 1 librados: calling monclient init 2024-03-25T19:39:15.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.882+0000 7f5053093640 10 monclient: init 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.883+0000 7f5053093640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.883+0000 7f5053093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 10 monclient(hunting): picked mon.noname-a con 0x7f504c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 10 monclient(hunting): picked mon.noname-c con 0x7f504c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 10 monclient(hunting): picked mon.noname-b con 0x7f504c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c1354c0 con 0x7f504c0a98f0 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c1319b0 con 0x7f504c1430c0 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c135070 con 0x7f504c13f710 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:15.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.884+0000 7f5053093640 10 monclient(hunting): authenticate will time out at 6432.638048s 2024-03-25T19:39:15.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.894+0000 7f5051630640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f504c1430c0 0x7f504c1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:46586/0 (socket says 172.21.15.29:46586) 2024-03-25T19:39:15.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.894+0000 7f5051630640 1 -- 172.21.15.29:0/2532794216 learned_addr learned my addr 172.21.15.29:0/2532794216 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:15.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.895+0000 7f50497fa640 10 client.?.objecter ms_handle_connect 0x7f504c1430c0 2024-03-25T19:39:15.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.895+0000 7f50497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.911+0000 7f50497fa640 10 client.?.objecter ms_handle_connect 0x7f504c13f710 2024-03-25T19:39:15.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.911+0000 7f50497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.919+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644626495 0 0) 0x7f504c1319b0 con 0x7f504c1430c0 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.920+0000 7f50497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.920+0000 7f50497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.920+0000 7f50497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.920+0000 7f50497fa640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:39:15.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.920+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030004eb0 con 0x7f504c1430c0 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829027230 0 0) 0x7f504c135070 con 0x7f504c13f710 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:39:15.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.924+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c1319b0 con 0x7f504c13f710 2024-03-25T19:39:15.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.926+0000 7f50497fa640 10 client.?.objecter ms_handle_connect 0x7f504c0a98f0 2024-03-25T19:39:15.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.926+0000 7f50497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564524153 0 0) 0x7f504c1319b0 con 0x7f504c13f710 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f504c135070 con 0x7f504c13f710 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f50380018b0 con 0x7f504c13f710 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 10 monclient(hunting): dump: 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:15.933 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:15.934 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:15.934 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.932+0000 7f50497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145503251 0 0) 0x7f504c1354c0 con 0x7f504c0a98f0 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:39:15.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.936+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c1319b0 con 0x7f504c0a98f0 2024-03-25T19:39:15.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774911038 0 0) 0x7f5030004eb0 con 0x7f504c1430c0 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f504c1354c0 con 0x7f504c1430c0 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5044001f40 con 0x7f504c1430c0 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.939+0000 7f50497fa640 10 monclient(hunting): dump: 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:15.940 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684869520 0 0) 0x7f504c135070 con 0x7f504c13f710 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 >> v1:172.21.15.196:6789/0 conn(0x7f504c1430c0 legacy=0x7f504c1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 >> v1:172.21.15.29:6789/0 conn(0x7f504c0a98f0 legacy=0x7f504c1d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 1 monclient: found mon.c 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c135200 con 0x7f504c13f710 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f50497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:15.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 5 monclient: authenticate success, global_id 4466 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 10 monclient: _renew_subs 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c0b7010 con 0x7f504c13f710 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 10 monclient: _renew_subs 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c13ba70 con 0x7f504c13f710 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.941+0000 7f5053093640 1 librados: init done 2024-03-25T19:39:15.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.942+0000 7f5053093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:15.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.942+0000 7f5053093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:15.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.942+0000 7f5053093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:15.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:15.942+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504c0b73b0 con 0x7f504c13f710 2024-03-25T19:39:16.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5038003610 con 0x7f504c13f710 2024-03-25T19:39:16.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f50380039a0 con 0x7f504c13f710 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 10 monclient: dump: 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:16.260 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f504affd640 4 set_mon_vals no callback set 2024-03-25T19:39:16.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.259+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f5038003bc0 con 0x7f504c13f710 2024-03-25T19:39:16.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.260+0000 7f5050e2f640 10 monclient: get_auth_request con 0x7f503006dfb0 auth_method 0 2024-03-25T19:39:16.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.260+0000 7f50497fa640 10 client.4466.objecter ms_handle_connect 0x7f503006dfb0 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f50380535c0 con 0x7f504c13f710 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 10 client.4466.objecter ms_dispatch 0x7f504c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.267+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f503804f0d0 con 0x7f504c13f710 2024-03-25T19:39:16.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.268+0000 7f50497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.268+0000 7f50497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:16.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.526+0000 7f5053093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.526+0000 7f5053093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.526+0000 7f5053093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:16.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.526+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f504c137160 con 0x7f504c13f710 2024-03-25T19:39:16.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.668+0000 7f50497fa640 1 -- 172.21.15.29:0/2532794216 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f5038051000 con 0x7f504c13f710 2024-03-25T19:39:16.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.668+0000 7f50497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.669 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.668+0000 7f50497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:16.669 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:16.669 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 10 librados: watch_flush enter 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 10 librados: watch_flush exit 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 10 client.4466.objecter successfully canceled tick 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 >> v1:172.21.15.29:6800/2314944989 conn(0x7f503006dfb0 legacy=0x7f5030070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 10 monclient: shutdown 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 >> v1:172.21.15.29:6790/0 conn(0x7f504c13f710 legacy=0x7f504c130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 shutdown_connections 2024-03-25T19:39:16.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 >> 172.21.15.29:0/2532794216 conn(0x7f504c0b7ea0 msgr2=0x7f504c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:16.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 shutdown_connections 2024-03-25T19:39:16.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 -- 172.21.15.29:0/2532794216 wait complete. 2024-03-25T19:39:16.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:16.671+0000 7f5053093640 1 librados: shutdown 2024-03-25T19:39:16.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:23.489 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.605+0000 7f1156dc4640 10 monclient: get_monmap_and_config 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.605+0000 7f1156dc4640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.605+0000 7f1156dc4640 10 monclient: monmap: 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:23.606430+0000 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:23.606430+0000 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:23.605 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:23.606 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:23.606 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.606+0000 7f1156dc4640 1 Processor -- start 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.606+0000 7f1156dc4640 1 -- start start 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.606+0000 7f1156dc4640 10 monclient: init 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f11500b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f11500b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f115014be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500af940 con 0x7f11500b4d60 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b0a70 con 0x7f115014be90 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b0d50 con 0x7f11500b1730 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.607+0000 7f1156dc4640 10 monclient(hunting): authenticate will time out at 6440.360331s 2024-03-25T19:39:23.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.616+0000 7f114ffff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f11500b1730 0x7f11500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:47388/0 (socket says 172.21.15.29:47388) 2024-03-25T19:39:23.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.616+0000 7f114ffff640 1 -- 172.21.15.29:0/2634659053 learned_addr learned my addr 172.21.15.29:0/2634659053 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:23.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710555877 0 0) 0x7f11500b0a70 con 0x7f115014be90 2024-03-25T19:39:23.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:39:23.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.628+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1140003520 con 0x7f115014be90 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647330438 0 0) 0x7f11500b0d50 con 0x7f11500b1730 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:39:23.638 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.638+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11500b0a70 con 0x7f11500b1730 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059920074 0 0) 0x7f1140003520 con 0x7f115014be90 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 >> v1:172.21.15.29:6790/0 conn(0x7f11500b1730 legacy=0x7f11500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 >> v1:172.21.15.29:6789/0 conn(0x7f11500b4d60 legacy=0x7f115013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500b1030 con 0x7f115014be90 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1148001a10 con 0x7f115014be90 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f114f7fe640 10 monclient: dump: 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f1156dc4640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:39:23.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.642+0000 7f1156dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1148001bf0 con 0x7f115014be90 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 1 -- 172.21.15.29:0/2634659053 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1148002000 con 0x7f115014be90 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f114f7fe640 10 monclient: dump: 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.650 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f1156dc4640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:23.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.650+0000 7f1156dc4640 4 set_mon_vals no callback set 2024-03-25T19:39:23.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.651+0000 7f1156dc4640 10 monclient: shutdown 2024-03-25T19:39:23.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.651+0000 7f1156dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:23.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.651+0000 7f1156dc4640 1 -- 172.21.15.29:0/2634659053 >> v1:172.21.15.196:6789/0 conn(0x7f115014be90 legacy=0x7f115014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.651+0000 7f1156dc4640 1 -- 172.21.15.29:0/2634659053 shutdown_connections 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.651+0000 7f1156dc4640 1 -- 172.21.15.29:0/2634659053 >> 172.21.15.29:0/2634659053 conn(0x7f11500617a0 msgr2=0x7f1150061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.652+0000 7f1156dc4640 1 -- 172.21.15.29:0/2634659053 shutdown_connections 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.652+0000 7f1156dc4640 1 -- 172.21.15.29:0/2634659053 wait complete. 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.652+0000 7f1156dc4640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.652+0000 7f1156dc4640 10 monclient: monmap: 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:23.653 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 librados: starting msgr at 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 librados: starting objecter 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 Processor -- start 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 -- start start 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 librados: setting wanted keys 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 1 librados: calling monclient init 2024-03-25T19:39:23.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.653+0000 7f1156dc4640 10 monclient: init 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f11500b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f11500b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f115014be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:23.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11501d79d0 con 0x7f11500b1730 2024-03-25T19:39:23.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11501d8600 con 0x7f115014be90 2024-03-25T19:39:23.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150138160 con 0x7f11500b4d60 2024-03-25T19:39:23.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.654+0000 7f1156dc4640 10 monclient(hunting): authenticate will time out at 6440.407366s 2024-03-25T19:39:23.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.667+0000 7f1154b60640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f11500b1730 0x7f11501d7ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48854/0 (socket says 172.21.15.29:48854) 2024-03-25T19:39:23.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.667+0000 7f1154b60640 1 -- 172.21.15.29:0/653839149 learned_addr learned my addr 172.21.15.29:0/653839149 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:23.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.667+0000 7f114d7fa640 10 client.?.objecter ms_handle_connect 0x7f11500b1730 2024-03-25T19:39:23.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.667+0000 7f114d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.674+0000 7f114d7fa640 10 client.?.objecter ms_handle_connect 0x7f115014be90 2024-03-25T19:39:23.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.674+0000 7f114d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.681+0000 7f114d7fa640 10 client.?.objecter ms_handle_connect 0x7f11500b4d60 2024-03-25T19:39:23.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.681+0000 7f114d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423828192 0 0) 0x7f11501d79d0 con 0x7f11500b1730 2024-03-25T19:39:23.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:39:23.682 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.682+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1134004eb0 con 0x7f11500b1730 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673552002 0 0) 0x7f11501d8600 con 0x7f115014be90 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:39:23.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.684+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11501d79d0 con 0x7f115014be90 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532572334 0 0) 0x7f1150138160 con 0x7f11500b4d60 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:39:23.701 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.701+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11501d8600 con 0x7f11500b4d60 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654145575 0 0) 0x7f1134004eb0 con 0x7f11500b1730 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1150138160 con 0x7f11500b1730 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f11440019d0 con 0x7f11500b1730 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): dump: 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.708 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1867728160 0 0) 0x7f11501d79d0 con 0x7f115014be90 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1134004eb0 con 0x7f115014be90 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1148001f00 con 0x7f115014be90 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.708+0000 7f114d7fa640 10 monclient(hunting): dump: 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.710 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928723527 0 0) 0x7f11501d8600 con 0x7f11500b4d60 2024-03-25T19:39:23.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11501d79d0 con 0x7f11500b4d60 2024-03-25T19:39:23.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f113c001930 con 0x7f11500b4d60 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.716+0000 7f114d7fa640 10 monclient(hunting): dump: 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.716 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2127399283 0 0) 0x7f11501d79d0 con 0x7f11500b4d60 2024-03-25T19:39:23.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 >> v1:172.21.15.196:6789/0 conn(0x7f115014be90 legacy=0x7f11501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 >> v1:172.21.15.29:6789/0 conn(0x7f11500b1730 legacy=0x7f11501d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 1 monclient: found mon.c 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11501382f0 con 0x7f11500b4d60 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f114d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 5 monclient: authenticate success, global_id 4472 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 10 monclient: _renew_subs 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11501433c0 con 0x7f11500b4d60 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 10 monclient: _renew_subs 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:23.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150140d00 con 0x7f11500b4d60 2024-03-25T19:39:23.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.724+0000 7f1156dc4640 1 librados: init done 2024-03-25T19:39:23.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.725+0000 7f1156dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.726+0000 7f1156dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.726+0000 7f1156dc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:23.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.726+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11500627b0 con 0x7f11500b4d60 2024-03-25T19:39:23.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f113c0047e0 con 0x7f11500b4d60 2024-03-25T19:39:23.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f113c001d80 con 0x7f11500b4d60 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114d7fa640 10 monclient: dump: 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:23.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:23.941+0000 7f114effd640 4 set_mon_vals no callback set 2024-03-25T19:39:24.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.045+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f113c01a960 con 0x7f11500b4d60 2024-03-25T19:39:24.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.046+0000 7f1154b60640 10 monclient: get_auth_request con 0x7f113406e160 auth_method 0 2024-03-25T19:39:24.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.046+0000 7f114d7fa640 10 client.4472.objecter ms_handle_connect 0x7f113406e160 2024-03-25T19:39:24.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f113c0546d0 con 0x7f11500b4d60 2024-03-25T19:39:24.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 10 client.4472.objecter ms_dispatch 0x7f1150000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f113c050010 con 0x7f11500b4d60 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:24.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.052+0000 7f114d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:24.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.296+0000 7f1156dc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.296+0000 7f1156dc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.296+0000 7f1156dc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:24.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.296+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1150141000 con 0x7f11500b4d60 2024-03-25T19:39:24.366 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.366+0000 7f114d7fa640 1 -- 172.21.15.29:0/653839149 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f113c052110 con 0x7f11500b4d60 2024-03-25T19:39:24.366 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.366+0000 7f114d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.366 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.366+0000 7f114d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:24.367 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:24.367 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 10 librados: watch_flush enter 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 10 librados: watch_flush exit 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 10 client.4472.objecter successfully canceled tick 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 >> v1:172.21.15.29:6800/2314944989 conn(0x7f113406e160 legacy=0x7f1134070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 10 monclient: shutdown 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 >> v1:172.21.15.29:6790/0 conn(0x7f11500b4d60 legacy=0x7f1150142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 shutdown_connections 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 >> 172.21.15.29:0/653839149 conn(0x7f11500617a0 msgr2=0x7f115013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 shutdown_connections 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 -- 172.21.15.29:0/653839149 wait complete. 2024-03-25T19:39:24.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:24.369+0000 7f1156dc4640 1 librados: shutdown 2024-03-25T19:39:24.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:30.340 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:00.504 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.398 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:00.561 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.437 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:00.601 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.454 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.616 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.462 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.624 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.467 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:00.631 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.475 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:00.639 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.488 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:00.650 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:30.504 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:00.667 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.186 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:31.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.299+0000 7fb9742b3640 10 monclient: get_monmap_and_config 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.299+0000 7fb9742b3640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.299+0000 7fb9742b3640 10 monclient: monmap: 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:31.300229+0000 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:31.300229+0000 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:31.299 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.300+0000 7fb9742b3640 1 Processor -- start 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.300+0000 7fb9742b3640 1 -- start start 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.300+0000 7fb9742b3640 10 monclient: init 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb96c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb96c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb96c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c142520 con 0x7fb96c13f710 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0b28a0 con 0x7fb96c1430c0 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0b2a30 con 0x7fb96c0a98f0 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.301+0000 7fb9742b3640 10 monclient(hunting): authenticate will time out at 6448.054078s 2024-03-25T19:39:31.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.306+0000 7fb972850640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb96c1430c0 0x7fb96c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:52244/0 (socket says 172.21.15.29:52244) 2024-03-25T19:39:31.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.306+0000 7fb972850640 1 -- 172.21.15.29:0/221201111 learned_addr learned my addr 172.21.15.29:0/221201111 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:31.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463823697 0 0) 0x7fb96c0b28a0 con 0x7fb96c1430c0 2024-03-25T19:39:31.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:39:31.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.315+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958003520 con 0x7fb96c1430c0 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121467355 0 0) 0x7fb96c142520 con 0x7fb96c13f710 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:39:31.320 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.320+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c0b28a0 con 0x7fb96c13f710 2024-03-25T19:39:31.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 219245596 0 0) 0x7fb96c0b28a0 con 0x7fb96c13f710 2024-03-25T19:39:31.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 >> v1:172.21.15.29:6790/0 conn(0x7fb96c0a98f0 legacy=0x7fb96c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 >> v1:172.21.15.196:6789/0 conn(0x7fb96c1430c0 legacy=0x7fb96c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 monclient: found mon.noname-a 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c0b44c0 con 0x7fb96c13f710 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb95c001670 con 0x7fb96c13f710 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb9742b3640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb9742b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb97104d640 10 monclient: dump: 2024-03-25T19:39:31.331 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.332 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.331+0000 7fb9742b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:31.352 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb95c0017d0 con 0x7fb96c13f710 2024-03-25T19:39:31.352 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.352 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 1 -- 172.21.15.29:0/221201111 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb95c001be0 con 0x7fb96c13f710 2024-03-25T19:39:31.352 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb9742b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:31.352 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb9742b3640 4 set_mon_vals no callback set 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.353+0000 7fb97104d640 10 monclient: dump: 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:31.353 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 10 monclient: shutdown 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 1 -- 172.21.15.29:0/221201111 >> v1:172.21.15.29:6789/0 conn(0x7fb96c13f710 legacy=0x7fb96c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 1 -- 172.21.15.29:0/221201111 shutdown_connections 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 1 -- 172.21.15.29:0/221201111 >> 172.21.15.29:0/221201111 conn(0x7fb96c0b7ea0 msgr2=0x7fb96c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 1 -- 172.21.15.29:0/221201111 shutdown_connections 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.354+0000 7fb9742b3640 1 -- 172.21.15.29:0/221201111 wait complete. 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 10 monclient: monmap: 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:31.355 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 librados: starting msgr at 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 librados: starting objecter 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 Processor -- start 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 -- start start 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 librados: setting wanted keys 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 1 librados: calling monclient init 2024-03-25T19:39:31.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.355+0000 7fb9742b3640 10 monclient: init 2024-03-25T19:39:31.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:31.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb96c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:31.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb96c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb96c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c1386e0 con 0x7fb96c0a98f0 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c137e80 con 0x7fb96c1430c0 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c1398f0 con 0x7fb96c13f710 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.356+0000 7fb9742b3640 10 monclient(hunting): authenticate will time out at 6448.109072s 2024-03-25T19:39:31.366 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.367+0000 7fb97204f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb96c0a98f0 0x7fb96c137770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43460/0 (socket says 172.21.15.29:43460) 2024-03-25T19:39:31.367 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.367+0000 7fb97204f640 1 -- 172.21.15.29:0/3767627188 learned_addr learned my addr 172.21.15.29:0/3767627188 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:31.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.369+0000 7fb962ffd640 10 client.?.objecter ms_handle_connect 0x7fb96c0a98f0 2024-03-25T19:39:31.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.369+0000 7fb962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.372+0000 7fb962ffd640 10 client.?.objecter ms_handle_connect 0x7fb96c13f710 2024-03-25T19:39:31.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.372+0000 7fb962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.377 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.377+0000 7fb962ffd640 10 client.?.objecter ms_handle_connect 0x7fb96c1430c0 2024-03-25T19:39:31.377 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.377+0000 7fb962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834405456 0 0) 0x7fb96c1398f0 con 0x7fb96c13f710 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:39:31.384 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.384+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb950004eb0 con 0x7fb96c13f710 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603051096 0 0) 0x7fb96c1386e0 con 0x7fb96c0a98f0 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:39:31.388 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.388+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c1398f0 con 0x7fb96c0a98f0 2024-03-25T19:39:31.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249185712 0 0) 0x7fb96c137e80 con 0x7fb96c1430c0 2024-03-25T19:39:31.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:39:31.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.391+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c1386e0 con 0x7fb96c1430c0 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897569765 0 0) 0x7fb950004eb0 con 0x7fb96c13f710 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb96c137e80 con 0x7fb96c13f710 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb95c001b20 con 0x7fb96c13f710 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.403 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.403+0000 7fb962ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559235111 0 0) 0x7fb96c1386e0 con 0x7fb96c1430c0 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb950004eb0 con 0x7fb96c1430c0 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb964001a30 con 0x7fb96c1430c0 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.416+0000 7fb962ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.416 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:31.417 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1859335236 0 0) 0x7fb96c137e80 con 0x7fb96c13f710 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 >> v1:172.21.15.196:6789/0 conn(0x7fb96c1430c0 legacy=0x7fb96c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 >> v1:172.21.15.29:6789/0 conn(0x7fb96c0a98f0 legacy=0x7fb96c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 1 monclient: found mon.c 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c1380a0 con 0x7fb96c13f710 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb9742b3640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb9742b3640 10 monclient: _renew_subs 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb9742b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.421+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c138350 con 0x7fb96c13f710 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.422+0000 7fb9742b3640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.422+0000 7fb9742b3640 10 monclient: _renew_subs 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.422+0000 7fb9742b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.422+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c1d7d80 con 0x7fb96c13f710 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.422+0000 7fb9742b3640 1 librados: init done 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.423+0000 7fb9742b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.423+0000 7fb9742b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.423+0000 7fb9742b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:31.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.423+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c1d8080 con 0x7fb96c13f710 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb95c0038a0 con 0x7fb96c13f710 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb95c003bb0 con 0x7fb96c13f710 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb962ffd640 10 monclient: dump: 2024-03-25T19:39:31.746 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:31.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.746+0000 7fb97084c640 4 set_mon_vals no callback set 2024-03-25T19:39:31.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.747+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fb95c019b80 con 0x7fb96c13f710 2024-03-25T19:39:31.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.747+0000 7fb97204f640 10 monclient: get_auth_request con 0x7fb95006df70 auth_method 0 2024-03-25T19:39:31.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.748+0000 7fb962ffd640 10 client.4478.objecter ms_handle_connect 0x7fb95006df70 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fb95c053700 con 0x7fb96c13f710 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 10 client.4478.objecter ms_dispatch 0x7fb96c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb95c04f210 con 0x7fb96c13f710 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.755+0000 7fb962ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:31.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.989+0000 7fb9742b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:31.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.989+0000 7fb9742b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:31.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.989+0000 7fb9742b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:39:31.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:31.989+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb96c138890 con 0x7fb96c13f710 2024-03-25T19:39:32.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.018+0000 7fb962ffd640 1 -- 172.21.15.29:0/3767627188 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fb96c138890 con 0x7fb96c13f710 2024-03-25T19:39:32.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.018+0000 7fb962ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.018+0000 7fb962ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:32.017 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:32.018 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 10 librados: watch_flush enter 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 10 librados: watch_flush exit 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 10 client.4478.objecter successfully canceled tick 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 >> v1:172.21.15.29:6800/2314944989 conn(0x7fb95006df70 legacy=0x7fb950070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 10 monclient: shutdown 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 >> v1:172.21.15.29:6790/0 conn(0x7fb96c13f710 legacy=0x7fb96c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 shutdown_connections 2024-03-25T19:39:32.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 >> 172.21.15.29:0/3767627188 conn(0x7fb96c0b7ea0 msgr2=0x7fb96c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:32.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 shutdown_connections 2024-03-25T19:39:32.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 -- 172.21.15.29:0/3767627188 wait complete. 2024-03-25T19:39:32.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:32.020+0000 7fb9742b3640 1 librados: shutdown 2024-03-25T19:39:32.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:38.837 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.956+0000 7f1273e06640 10 monclient: get_monmap_and_config 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.956+0000 7f1273e06640 10 monclient: build_initial_monmap 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.956+0000 7f1273e06640 10 monclient: monmap: 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:38.957270+0000 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:38.957270+0000 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:38.956 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.957+0000 7f1273e06640 1 Processor -- start 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.957+0000 7f1273e06640 1 -- start start 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.957+0000 7f1273e06640 10 monclient: init 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-b con 0x7f126c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-a con 0x7f126c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-c con 0x7f126c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c142520 con 0x7f126c13f710 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c0b28a0 con 0x7f126c1430c0 2024-03-25T19:39:38.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c0b2a30 con 0x7f126c0a98f0 2024-03-25T19:39:38.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:38.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.958+0000 7f1273e06640 10 monclient(hunting): authenticate will time out at 6455.711029s 2024-03-25T19:39:38.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.974+0000 7f12713a1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f126c13f710 0x7f126c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42184/0 (socket says 172.21.15.29:42184) 2024-03-25T19:39:38.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.974+0000 7f12713a1640 1 -- 172.21.15.29:0/1938939989 learned_addr learned my addr 172.21.15.29:0/1938939989 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:38.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848702656 0 0) 0x7f126c0b28a0 con 0x7f126c1430c0 2024-03-25T19:39:38.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 10 monclient(hunting): _negotiate 2024-03-25T19:39:38.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:38.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:38.990 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:39:38.990 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.990+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1258003520 con 0x7f126c1430c0 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062453353 0 0) 0x7f126c0b2a30 con 0x7f126c0a98f0 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 10 monclient(hunting): _negotiate 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:39:38.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:38.996+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c0b28a0 con 0x7f126c0a98f0 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190127387 0 0) 0x7f1258003520 con 0x7f126c1430c0 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 >> v1:172.21.15.29:6790/0 conn(0x7f126c0a98f0 legacy=0x7f126c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 >> v1:172.21.15.29:6789/0 conn(0x7f126c13f710 legacy=0x7f126c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 monclient: found mon.noname-c 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c0b44c0 con 0x7f126c1430c0 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1264001a10 con 0x7f126c1430c0 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1273e06640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1273e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1270ba0640 10 monclient: dump: 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.003 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.002+0000 7f1273e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1264001b70 con 0x7f126c1430c0 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 1 -- 172.21.15.29:0/1938939989 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1264001f80 con 0x7f126c1430c0 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1270ba0640 10 monclient: dump: 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.018 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.019 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.019 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.019 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.019 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1273e06640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:39.019 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.018+0000 7f1273e06640 4 set_mon_vals no callback set 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 10 monclient: shutdown 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 1 -- 172.21.15.29:0/1938939989 >> v1:172.21.15.196:6789/0 conn(0x7f126c1430c0 legacy=0x7f126c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 1 -- 172.21.15.29:0/1938939989 shutdown_connections 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 1 -- 172.21.15.29:0/1938939989 >> 172.21.15.29:0/1938939989 conn(0x7f126c0b7ea0 msgr2=0x7f126c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 1 -- 172.21.15.29:0/1938939989 shutdown_connections 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.019+0000 7f1273e06640 1 -- 172.21.15.29:0/1938939989 wait complete. 2024-03-25T19:39:39.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 10 monclient: build_initial_monmap 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 10 monclient: monmap: 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 librados: starting msgr at 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 librados: starting objecter 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 Processor -- start 2024-03-25T19:39:39.021 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 -- start start 2024-03-25T19:39:39.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 librados: setting wanted keys 2024-03-25T19:39:39.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 1 librados: calling monclient init 2024-03-25T19:39:39.022 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.020+0000 7f1273e06640 10 monclient: init 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-a con 0x7f126c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-c con 0x7f126c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 10 monclient(hunting): picked mon.noname-b con 0x7f126c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.021+0000 7f1273e06640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c0abd30 con 0x7f126c0a98f0 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.022+0000 7f1273e06640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c1d7810 con 0x7f126c1430c0 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.022+0000 7f1273e06640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c1d8440 con 0x7f126c13f710 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.022+0000 7f1273e06640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:39.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.022+0000 7f1273e06640 10 monclient(hunting): authenticate will time out at 6455.774980s 2024-03-25T19:39:39.031 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.032+0000 7f12723a3640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f126c1430c0 0x7f126c1383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:38010/0 (socket says 172.21.15.29:38010) 2024-03-25T19:39:39.032 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.032+0000 7f12723a3640 1 -- 172.21.15.29:0/257353172 learned_addr learned my addr 172.21.15.29:0/257353172 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:39.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.035+0000 7f12627fc640 10 client.?.objecter ms_handle_connect 0x7f126c1430c0 2024-03-25T19:39:39.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.035+0000 7f12627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.039+0000 7f12627fc640 10 client.?.objecter ms_handle_connect 0x7f126c0a98f0 2024-03-25T19:39:39.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.039+0000 7f12627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.039+0000 7f12627fc640 10 client.?.objecter ms_handle_connect 0x7f126c13f710 2024-03-25T19:39:39.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.039+0000 7f12627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521040816 0 0) 0x7f126c1d7810 con 0x7f126c1430c0 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:39:39.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.052+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250004eb0 con 0x7f126c1430c0 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100632604 0 0) 0x7f126c0abd30 con 0x7f126c0a98f0 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:39:39.054 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.054+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c1d7810 con 0x7f126c0a98f0 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3369085857 0 0) 0x7f1250004eb0 con 0x7f126c1430c0 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f126c0abd30 con 0x7f126c1430c0 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1264001f80 con 0x7f126c1430c0 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 10 monclient(hunting): dump: 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.064 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.064+0000 7f12627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.065+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596488666 0 0) 0x7f126c1d8440 con 0x7f126c13f710 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.066+0000 7f12627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.066+0000 7f12627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.066+0000 7f12627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.066+0000 7f12627fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:39:39.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.066+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250004eb0 con 0x7f126c13f710 2024-03-25T19:39:39.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3479063052 0 0) 0x7f126c1d7810 con 0x7f126c0a98f0 2024-03-25T19:39:39.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f126c1d8440 con 0x7f126c0a98f0 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1268001930 con 0x7f126c0a98f0 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.079+0000 7f12627fc640 10 monclient(hunting): dump: 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.079 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.081+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231397715 0 0) 0x7f1250004eb0 con 0x7f126c13f710 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.081+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f126c1d7810 con 0x7f126c13f710 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.081+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f125c001860 con 0x7f126c13f710 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.082+0000 7f12627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.082+0000 7f12627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:39.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.082+0000 7f12627fc640 10 monclient(hunting): dump: 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.082 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.085+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250319817 0 0) 0x7f126c0abd30 con 0x7f126c1430c0 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 >> v1:172.21.15.29:6790/0 conn(0x7f126c13f710 legacy=0x7f126c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 >> v1:172.21.15.29:6789/0 conn(0x7f126c0a98f0 legacy=0x7f126c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 1 monclient: found mon.b 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c1d5000 con 0x7f126c1430c0 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f12627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:39:39.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 10 monclient: _renew_subs 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c1d5250 con 0x7f126c1430c0 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 10 monclient: _renew_subs 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c1d43c0 con 0x7f126c1430c0 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.086+0000 7f1273e06640 1 librados: init done 2024-03-25T19:39:39.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.087+0000 7f1273e06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.087+0000 7f1273e06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.087+0000 7f1273e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.087+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c1d7670 con 0x7f126c1430c0 2024-03-25T19:39:39.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.093+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1264003c20 con 0x7f126c1430c0 2024-03-25T19:39:39.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.093+0000 7f12627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.093+0000 7f1263fff640 4 set_mon_vals no callback set 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.358+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f1264003dd0 con 0x7f126c1430c0 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.358+0000 7f12627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.358+0000 7f12627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.358+0000 7f12627fc640 10 monclient: dump: 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:39.358 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:39.359 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:39.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.359+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f1264003fb0 con 0x7f126c1430c0 2024-03-25T19:39:39.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.360+0000 7f1271ba2640 10 monclient: get_auth_request con 0x7f125006e160 auth_method 0 2024-03-25T19:39:39.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.360+0000 7f12627fc640 10 client.4483.objecter ms_handle_connect 0x7f125006e160 2024-03-25T19:39:39.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.361+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f1264054a60 con 0x7f126c1430c0 2024-03-25T19:39:39.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.361+0000 7f12627fc640 10 client.4483.objecter ms_dispatch 0x7f126c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:39.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.361+0000 7f12627fc640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:39.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.361+0000 7f12627fc640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:39.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.361+0000 7f12627fc640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:39:39.364 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.365+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1264050570 con 0x7f126c1430c0 2024-03-25T19:39:39.364 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.365+0000 7f12627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.364 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.365+0000 7f12627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:39.594 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.595+0000 7f1273e06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.595+0000 7f1273e06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.595+0000 7f1273e06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:39:39.595 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.595+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f126c1d7a50 con 0x7f126c1430c0 2024-03-25T19:39:39.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.613+0000 7f12627fc640 1 -- 172.21.15.29:0/257353172 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f126c1d7a50 con 0x7f126c1430c0 2024-03-25T19:39:39.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.613+0000 7f12627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.613+0000 7f12627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:39.613 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:39.613 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 10 librados: watch_flush enter 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 10 librados: watch_flush exit 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 >> v1:172.21.15.29:6800/2314944989 conn(0x7f125006e160 legacy=0x7f1250070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 10 monclient: shutdown 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 >> v1:172.21.15.196:6789/0 conn(0x7f126c1430c0 legacy=0x7f126c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 shutdown_connections 2024-03-25T19:39:39.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 >> 172.21.15.29:0/257353172 conn(0x7f126c0b7ea0 msgr2=0x7f126c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 shutdown_connections 2024-03-25T19:39:39.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 -- 172.21.15.29:0/257353172 wait complete. 2024-03-25T19:39:39.620 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:39.616+0000 7f1273e06640 1 librados: shutdown 2024-03-25T19:39:39.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:46.434 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:46.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.550+0000 7f9dbe820640 10 monclient: get_monmap_and_config 2024-03-25T19:39:46.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.550+0000 7f9dbe820640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.550+0000 7f9dbe820640 10 monclient: monmap: 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:46.551190+0000 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:46.551190+0000 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 1 Processor -- start 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 1 -- start start 2024-03-25T19:39:46.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient: init 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-b con 0x7f9db80a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-c con 0x7f9db813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-a con 0x7f9db81430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db8142520 con 0x7f9db81430c0 2024-03-25T19:39:46.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.551+0000 7f9dbe820640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80b28a0 con 0x7f9db813f710 2024-03-25T19:39:46.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.552+0000 7f9dbe820640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80b2a30 con 0x7f9db80a98f0 2024-03-25T19:39:46.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.552+0000 7f9dbe820640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.552+0000 7f9dbe820640 10 monclient(hunting): authenticate will time out at 6463.304877s 2024-03-25T19:39:46.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.557+0000 7f9dbcdbd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9db81430c0 0x7f9db814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43034/0 (socket says 172.21.15.29:43034) 2024-03-25T19:39:46.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.557+0000 7f9dbcdbd640 1 -- 172.21.15.29:0/604574146 learned_addr learned my addr 172.21.15.29:0/604574146 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537844910 0 0) 0x7f9db8142520 con 0x7f9db81430c0 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:39:46.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.569+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da0003520 con 0x7f9db81430c0 2024-03-25T19:39:46.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455459851 0 0) 0x7f9da0003520 con 0x7f9db81430c0 2024-03-25T19:39:46.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 >> v1:172.21.15.29:6790/0 conn(0x7f9db80a98f0 legacy=0x7f9db80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 >> v1:172.21.15.196:6789/0 conn(0x7f9db813f710 legacy=0x7f9db81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80b44c0 con 0x7f9db81430c0 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9db0001a10 con 0x7f9db81430c0 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9db6ffd640 10 monclient: dump: 2024-03-25T19:39:46.575 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9dbe820640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:39:46.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.574+0000 7f9dbe820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:46.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.599+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9db0001b70 con 0x7f9db81430c0 2024-03-25T19:39:46.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.599+0000 7f9db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:46.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.599+0000 7f9db6ffd640 1 -- 172.21.15.29:0/604574146 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9db0001f80 con 0x7f9db81430c0 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.599+0000 7f9db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9db6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9db6ffd640 10 monclient: dump: 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:46.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 4 set_mon_vals no callback set 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 10 monclient: shutdown 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 1 -- 172.21.15.29:0/604574146 >> v1:172.21.15.29:6789/0 conn(0x7f9db81430c0 legacy=0x7f9db814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 1 -- 172.21.15.29:0/604574146 shutdown_connections 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 1 -- 172.21.15.29:0/604574146 >> 172.21.15.29:0/604574146 conn(0x7f9db80b7ea0 msgr2=0x7f9db80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 1 -- 172.21.15.29:0/604574146 shutdown_connections 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.600+0000 7f9dbe820640 1 -- 172.21.15.29:0/604574146 wait complete. 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.601+0000 7f9dbe820640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.601+0000 7f9dbe820640 10 monclient: monmap: 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:46.603 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.601+0000 7f9dbe820640 1 librados: starting msgr at 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.601+0000 7f9dbe820640 1 librados: starting objecter 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.602+0000 7f9dbe820640 1 Processor -- start 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.602+0000 7f9dbe820640 1 -- start start 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.602+0000 7f9dbe820640 1 librados: setting wanted keys 2024-03-25T19:39:46.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.602+0000 7f9dbe820640 1 librados: calling monclient init 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.602+0000 7f9dbe820640 10 monclient: init 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-b con 0x7f9db80a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-c con 0x7f9db813f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient(hunting): picked mon.noname-a con 0x7f9db81430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db8139a10 con 0x7f9db81430c0 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db81d7880 con 0x7f9db80a98f0 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db8137890 con 0x7f9db813f710 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.603+0000 7f9dbe820640 10 monclient(hunting): authenticate will time out at 6463.356131s 2024-03-25T19:39:46.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.612+0000 7f9db7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9db80a98f0 0x7f9db81d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:37110/0 (socket says 172.21.15.29:37110) 2024-03-25T19:39:46.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.612+0000 7f9db7fff640 1 -- 172.21.15.29:0/3342016855 learned_addr learned my addr 172.21.15.29:0/3342016855 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:46.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.613+0000 7f9db4ff9640 10 client.?.objecter ms_handle_connect 0x7f9db80a98f0 2024-03-25T19:39:46.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.613+0000 7f9db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.617+0000 7f9db4ff9640 10 client.?.objecter ms_handle_connect 0x7f9db813f710 2024-03-25T19:39:46.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.617+0000 7f9db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.623+0000 7f9db4ff9640 10 client.?.objecter ms_handle_connect 0x7f9db81430c0 2024-03-25T19:39:46.623 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.623+0000 7f9db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49178378 0 0) 0x7f9db81d7880 con 0x7f9db80a98f0 2024-03-25T19:39:46.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:39:46.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.626+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da4004eb0 con 0x7f9db80a98f0 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086658669 0 0) 0x7f9db8139a10 con 0x7f9db81430c0 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:39:46.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.634+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da40051d0 con 0x7f9db81430c0 2024-03-25T19:39:46.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.651+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338663961 0 0) 0x7f9da4004eb0 con 0x7f9db80a98f0 2024-03-25T19:39:46.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.651+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9db8139a10 con 0x7f9db80a98f0 2024-03-25T19:39:46.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.651+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9da8001990 con 0x7f9db80a98f0 2024-03-25T19:39:46.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.652+0000 7f9db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.652+0000 7f9db4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.652+0000 7f9db4ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.652+0000 7f9db4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:46.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121063799 0 0) 0x7f9db8137890 con 0x7f9db813f710 2024-03-25T19:39:46.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:39:46.658 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.658+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da4004eb0 con 0x7f9db813f710 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1741224723 0 0) 0x7f9da40051d0 con 0x7f9db81430c0 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9db8137890 con 0x7f9db81430c0 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9db0001e80 con 0x7f9db81430c0 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.659+0000 7f9db4ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.659 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:46.660 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.670+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 679494605 0 0) 0x7f9db8137890 con 0x7f9db81430c0 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 >> v1:172.21.15.29:6790/0 conn(0x7f9db813f710 legacy=0x7f9db8138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 >> v1:172.21.15.196:6789/0 conn(0x7f9db80a98f0 legacy=0x7f9db81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 1 monclient: found mon.a 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db81d84b0 con 0x7f9db81430c0 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9db4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 10 monclient: _renew_subs 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db81d51f0 con 0x7f9db81430c0 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 10 monclient: _renew_subs 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db8138c60 con 0x7f9db81430c0 2024-03-25T19:39:46.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.671+0000 7f9dbe820640 1 librados: init done 2024-03-25T19:39:46.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.672+0000 7f9dbe820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.672+0000 7f9dbe820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.672+0000 7f9dbe820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:46.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.672+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db80627b0 con 0x7f9db81430c0 2024-03-25T19:39:46.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9db0002040 con 0x7f9db81430c0 2024-03-25T19:39:46.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:46.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9db0004fb0 con 0x7f9db81430c0 2024-03-25T19:39:46.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db4ff9640 10 monclient: dump: 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:46.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.995+0000 7f9db67fc640 4 set_mon_vals no callback set 2024-03-25T19:39:46.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.996+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9db001af80 con 0x7f9db81430c0 2024-03-25T19:39:46.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.997+0000 7f9db7fff640 10 monclient: get_auth_request con 0x7f9da406e160 auth_method 0 2024-03-25T19:39:46.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:46.997+0000 7f9db4ff9640 10 client.4491.objecter ms_handle_connect 0x7f9da406e160 2024-03-25T19:39:47.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.002+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f9db0054ae0 con 0x7f9db81430c0 2024-03-25T19:39:47.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.002+0000 7f9db4ff9640 10 client.4491.objecter ms_dispatch 0x7f9db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:47.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.002+0000 7f9db4ff9640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:47.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.002+0000 7f9db4ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:47.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.003+0000 7f9db4ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T19:39:47.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.003+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9db00505f0 con 0x7f9db81430c0 2024-03-25T19:39:47.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.003+0000 7f9db4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:47.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.003+0000 7f9db4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:47.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.245+0000 7f9dbe820640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.245+0000 7f9dbe820640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.246+0000 7f9dbe820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:47.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.246+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9db813df90 con 0x7f9db81430c0 2024-03-25T19:39:47.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.277+0000 7f9db4ff9640 1 -- 172.21.15.29:0/3342016855 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f9db0052520 con 0x7f9db81430c0 2024-03-25T19:39:47.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.277+0000 7f9db4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.277+0000 7f9db4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:47.277 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:47.277 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:47.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 10 librados: watch_flush enter 2024-03-25T19:39:47.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 10 librados: watch_flush exit 2024-03-25T19:39:47.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T19:39:47.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 10 client.4491.objecter successfully canceled tick 2024-03-25T19:39:47.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 >> v1:172.21.15.29:6800/2314944989 conn(0x7f9da406e160 legacy=0x7f9da4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 10 monclient: shutdown 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 >> v1:172.21.15.29:6789/0 conn(0x7f9db81430c0 legacy=0x7f9db81e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 shutdown_connections 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 >> 172.21.15.29:0/3342016855 conn(0x7f9db80b7ea0 msgr2=0x7f9db8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 shutdown_connections 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 -- 172.21.15.29:0/3342016855 wait complete. 2024-03-25T19:39:47.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:47.279+0000 7f9dbe820640 1 librados: shutdown 2024-03-25T19:39:47.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:50.408 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:20.571 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.468 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:20.620 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.468 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:20.615 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.468 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:20.617 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.494 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:20.656 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.499 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:20.663 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.519 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:20.682 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.524 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:20.688 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:50.552 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:20.715 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:54.097 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:54.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.212+0000 7efdc8513640 10 monclient: get_monmap_and_config 2024-03-25T19:39:54.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.212+0000 7efdc8513640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.212+0000 7efdc8513640 10 monclient: monmap: 2024-03-25T19:39:54.212 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:39:54.213707+0000 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:39:54.213707+0000 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.213 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:54.214 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:39:54.214 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:39:54.214 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.213+0000 7efdc8513640 1 Processor -- start 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.213+0000 7efdc8513640 1 -- start start 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.213+0000 7efdc8513640 10 monclient: init 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.214+0000 7efdc8513640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-a con 0x7efdc00a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-c con 0x7efdc013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-b con 0x7efdc01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:54.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0142520 con 0x7efdc00a98f0 2024-03-25T19:39:54.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00b28a0 con 0x7efdc013f710 2024-03-25T19:39:54.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00b2a30 con 0x7efdc01430c0 2024-03-25T19:39:54.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.215+0000 7efdc8513640 10 monclient(hunting): authenticate will time out at 6470.968092s 2024-03-25T19:39:54.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.231+0000 7efdc6ab0640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7efdc01430c0 0x7efdc014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:58128/0 (socket says 172.21.15.29:58128) 2024-03-25T19:39:54.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.231+0000 7efdc6ab0640 1 -- 172.21.15.29:0/1352829090 learned_addr learned my addr 172.21.15.29:0/1352829090 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:54.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134215086 0 0) 0x7efdc00b2a30 con 0x7efdc01430c0 2024-03-25T19:39:54.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:39:54.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.245+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac003520 con 0x7efdc01430c0 2024-03-25T19:39:54.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785164084 0 0) 0x7efdc00b28a0 con 0x7efdc013f710 2024-03-25T19:39:54.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.246 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:39:54.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.247+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc00b2a30 con 0x7efdc013f710 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884835029 0 0) 0x7efdc0142520 con 0x7efdc00a98f0 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:39:54.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.252+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc00b28a0 con 0x7efdc00a98f0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456519731 0 0) 0x7efdc00b28a0 con 0x7efdc00a98f0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 >> v1:172.21.15.29:6790/0 conn(0x7efdc01430c0 legacy=0x7efdc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 >> v1:172.21.15.196:6789/0 conn(0x7efdc013f710 legacy=0x7efdc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 monclient: found mon.noname-a 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00b44c0 con 0x7efdc00a98f0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdbc0015b0 con 0x7efdc00a98f0 2024-03-25T19:39:54.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc52ad640 10 monclient: dump: 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc8513640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:39:54.262 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.261+0000 7efdc8513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdbc001710 con 0x7efdc00a98f0 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 1 -- 172.21.15.29:0/1352829090 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdbc001b60 con 0x7efdc00a98f0 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc52ad640 10 monclient: dump: 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.266 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc8513640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:54.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.266+0000 7efdc8513640 4 set_mon_vals no callback set 2024-03-25T19:39:54.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 10 monclient: shutdown 2024-03-25T19:39:54.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:54.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 1 -- 172.21.15.29:0/1352829090 >> v1:172.21.15.29:6789/0 conn(0x7efdc00a98f0 legacy=0x7efdc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 1 -- 172.21.15.29:0/1352829090 shutdown_connections 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 1 -- 172.21.15.29:0/1352829090 >> 172.21.15.29:0/1352829090 conn(0x7efdc00b7ea0 msgr2=0x7efdc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 1 -- 172.21.15.29:0/1352829090 shutdown_connections 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.267+0000 7efdc8513640 1 -- 172.21.15.29:0/1352829090 wait complete. 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.268+0000 7efdc8513640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.268+0000 7efdc8513640 10 monclient: monmap: 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:39:54.269 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.268+0000 7efdc8513640 1 librados: starting msgr at 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.268+0000 7efdc8513640 1 librados: starting objecter 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 Processor -- start 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 -- start start 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 librados: setting wanted keys 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 librados: calling monclient init 2024-03-25T19:39:54.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient: init 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-c con 0x7efdc00a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-b con 0x7efdc013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient(hunting): picked mon.noname-a con 0x7efdc01430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0136830 con 0x7efdc01430c0 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0137e40 con 0x7efdc013f710 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0138120 con 0x7efdc00a98f0 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.271 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.269+0000 7efdc8513640 10 monclient(hunting): authenticate will time out at 6471.022665s 2024-03-25T19:39:54.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.288+0000 7efdc5aae640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7efdc013f710 0x7efdc0138590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41756/0 (socket says 172.21.15.29:41756) 2024-03-25T19:39:54.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.288+0000 7efdc5aae640 1 -- 172.21.15.29:0/1864467610 learned_addr learned my addr 172.21.15.29:0/1864467610 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:39:54.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.289+0000 7efdb6ffd640 10 client.?.objecter ms_handle_connect 0x7efdc013f710 2024-03-25T19:39:54.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.289+0000 7efdb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.301+0000 7efdb6ffd640 10 client.?.objecter ms_handle_connect 0x7efdc00a98f0 2024-03-25T19:39:54.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.301+0000 7efdb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.302+0000 7efdb6ffd640 10 client.?.objecter ms_handle_connect 0x7efdc01430c0 2024-03-25T19:39:54.301 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.302+0000 7efdb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814480660 0 0) 0x7efdc0137e40 con 0x7efdc013f710 2024-03-25T19:39:54.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:39:54.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.310+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4004eb0 con 0x7efdc013f710 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581930360 0 0) 0x7efdc0138120 con 0x7efdc00a98f0 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:39:54.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.316+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc0137e40 con 0x7efdc00a98f0 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048069471 0 0) 0x7efdc0136830 con 0x7efdc01430c0 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:39:54.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.321+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc0138120 con 0x7efdc01430c0 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742132389 0 0) 0x7efdc0138120 con 0x7efdc01430c0 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc0136830 con 0x7efdc01430c0 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdb8001a30 con 0x7efdc01430c0 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:54.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.331+0000 7efdb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:54.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754673666 0 0) 0x7efda4004eb0 con 0x7efdc013f710 2024-03-25T19:39:54.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc0138120 con 0x7efdc013f710 2024-03-25T19:39:54.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdb0001a10 con 0x7efdc013f710 2024-03-25T19:39:54.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.333+0000 7efdb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.336 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 699888941 0 0) 0x7efdc0137e40 con 0x7efdc00a98f0 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda4004eb0 con 0x7efdc00a98f0 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdbc001a20 con 0x7efdc00a98f0 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:54.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.337+0000 7efdb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.338 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3034526132 0 0) 0x7efdc0136830 con 0x7efdc01430c0 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 >> v1:172.21.15.29:6790/0 conn(0x7efdc00a98f0 legacy=0x7efdc01db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 >> v1:172.21.15.196:6789/0 conn(0x7efdc013f710 legacy=0x7efdc0138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 1 monclient: found mon.a 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00abac0 con 0x7efdc01430c0 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 10 monclient: _renew_subs 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc01db7f0 con 0x7efdc01430c0 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 10 monclient: _renew_subs 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc0137a10 con 0x7efdc01430c0 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.346+0000 7efdc8513640 1 librados: init done 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.347+0000 7efdc8513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.347+0000 7efdc8513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.347+0000 7efdc8513640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.347+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc00627b0 con 0x7efdc01430c0 2024-03-25T19:39:54.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.557+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb8001f40 con 0x7efdc01430c0 2024-03-25T19:39:54.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.557+0000 7efdb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.557 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.557+0000 7efdc4aac640 4 set_mon_vals no callback set 2024-03-25T19:39:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.652+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7efdb8004d00 con 0x7efdc01430c0 2024-03-25T19:39:54.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.652+0000 7efdb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.652+0000 7efdb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.652+0000 7efdb6ffd640 10 monclient: dump: 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:39:54.652 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:39:54.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.653+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7efdb801acd0 con 0x7efdc01430c0 2024-03-25T19:39:54.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.653+0000 7efdc62af640 10 monclient: get_auth_request con 0x7efda406e0f0 auth_method 0 2024-03-25T19:39:54.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.653+0000 7efdb6ffd640 10 client.4497.objecter ms_handle_connect 0x7efda406e0f0 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7efdb8054980 con 0x7efdc01430c0 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 10 client.4497.objecter ms_dispatch 0x7efdc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdb80502c0 con 0x7efdc01430c0 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.659+0000 7efdb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:54.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.886+0000 7efdc8513640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:54.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.886+0000 7efdc8513640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:54.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.886+0000 7efdc8513640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:39:54.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.886+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc0069f40 con 0x7efdc01430c0 2024-03-25T19:39:54.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.921+0000 7efdb6ffd640 1 -- 172.21.15.29:0/1864467610 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7efdb80523c0 con 0x7efdc01430c0 2024-03-25T19:39:54.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.921+0000 7efdb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:54.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.921+0000 7efdb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:54.922 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:39:54.922 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 10 librados: watch_flush enter 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 10 librados: watch_flush exit 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 >> v1:172.21.15.29:6800/2314944989 conn(0x7efda406e0f0 legacy=0x7efda40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 10 monclient: shutdown 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 >> v1:172.21.15.29:6789/0 conn(0x7efdc01430c0 legacy=0x7efdc01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 shutdown_connections 2024-03-25T19:39:54.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 >> 172.21.15.29:0/1864467610 conn(0x7efdc00b7ea0 msgr2=0x7efdc013d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:54.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 shutdown_connections 2024-03-25T19:39:54.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 -- 172.21.15.29:0/1864467610 wait complete. 2024-03-25T19:39:54.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:39:54.924+0000 7efdc8513640 1 librados: shutdown 2024-03-25T19:39:54.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:01.743 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.860+0000 7f3f0ca16640 10 monclient: get_monmap_and_config 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.860+0000 7f3f0ca16640 10 monclient: build_initial_monmap 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.860+0000 7f3f0ca16640 10 monclient: monmap: 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:01.860 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:01.861402+0000 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:01.861402+0000 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:01.861 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.861+0000 7f3f0ca16640 1 Processor -- start 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.861+0000 7f3f0ca16640 1 -- start start 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.861+0000 7f3f0ca16640 10 monclient: init 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-b con 0x7f3f080a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-c con 0x7f3f081430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f08142520 con 0x7f3f0813f710 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f080b28a0 con 0x7f3f081430c0 2024-03-25T19:40:01.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f080b2a30 con 0x7f3f080a98f0 2024-03-25T19:40:01.864 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:01.864 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.862+0000 7f3f0ca16640 10 monclient(hunting): authenticate will time out at 6478.615338s 2024-03-25T19:40:01.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.870+0000 7f3f06d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3f081430c0 0x7f3f0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:60306/0 (socket says 172.21.15.29:60306) 2024-03-25T19:40:01.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.870+0000 7f3f06d9d640 1 -- 172.21.15.29:0/1311860443 learned_addr learned my addr 172.21.15.29:0/1311860443 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:01.885 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477452899 0 0) 0x7f3f080b2a30 con 0x7f3f080a98f0 2024-03-25T19:40:01.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:01.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:01.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:01.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:40:01.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.886+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef0003520 con 0x7f3f080a98f0 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021461523 0 0) 0x7f3f080b28a0 con 0x7f3f081430c0 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:40:01.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.888+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f080b2a30 con 0x7f3f081430c0 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1144641767 0 0) 0x7f3ef0003520 con 0x7f3f080a98f0 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 >> v1:172.21.15.196:6789/0 conn(0x7f3f081430c0 legacy=0x7f3f0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 >> v1:172.21.15.29:6789/0 conn(0x7f3f0813f710 legacy=0x7f3f081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 monclient: found mon.noname-b 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080b44c0 con 0x7f3f080a98f0 2024-03-25T19:40:01.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: _finish_auth 0 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3ef8001690 con 0x7f3f080a98f0 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0ca16640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0ca16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0559a640 10 monclient: dump: 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:01.903 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:01.904 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:01.904 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.902+0000 7f3f0ca16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ef80017f0 con 0x7f3f080a98f0 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 1 -- 172.21.15.29:0/1311860443 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3ef8001c00 con 0x7f3f080a98f0 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0559a640 10 monclient: dump: 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:01.909 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0ca16640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:01.910 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.909+0000 7f3f0ca16640 4 set_mon_vals no callback set 2024-03-25T19:40:01.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 10 monclient: shutdown 2024-03-25T19:40:01.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:01.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1311860443 >> v1:172.21.15.29:6790/0 conn(0x7f3f080a98f0 legacy=0x7f3f080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1311860443 shutdown_connections 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1311860443 >> 172.21.15.29:0/1311860443 conn(0x7f3f080b7ea0 msgr2=0x7f3f080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1311860443 shutdown_connections 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.910+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1311860443 wait complete. 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.911+0000 7f3f0ca16640 10 monclient: build_initial_monmap 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.911+0000 7f3f0ca16640 10 monclient: monmap: 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:01.912 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.911+0000 7f3f0ca16640 1 librados: starting msgr at 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.911+0000 7f3f0ca16640 1 librados: starting objecter 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 1 Processor -- start 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 1 -- start start 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 1 librados: setting wanted keys 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 1 librados: calling monclient init 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 10 monclient: init 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:01.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.912+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-a con 0x7f3f080a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-c con 0x7f3f0813f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 10 monclient(hunting): picked mon.noname-b con 0x7f3f081430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f081db7d0 con 0x7f3f080a98f0 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f080b1420 con 0x7f3f081430c0 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f080b15b0 con 0x7f3f0813f710 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:01.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.913+0000 7f3f0ca16640 10 monclient(hunting): authenticate will time out at 6478.665920s 2024-03-25T19:40:01.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.921+0000 7f3f05d9b640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f3f0813f710 0x7f3f080b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:38008/0 (socket says 172.21.15.29:38008) 2024-03-25T19:40:01.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.922+0000 7f3f05d9b640 1 -- 172.21.15.29:0/1687134144 learned_addr learned my addr 172.21.15.29:0/1687134144 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:01.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.925+0000 7f3ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f3f080a98f0 2024-03-25T19:40:01.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.925+0000 7f3ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:01.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.926+0000 7f3ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f3f0813f710 2024-03-25T19:40:01.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.926+0000 7f3ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:01.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.926+0000 7f3ef6ffd640 10 client.?.objecter ms_handle_connect 0x7f3f081430c0 2024-03-25T19:40:01.926 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.926+0000 7f3ef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617925521 0 0) 0x7f3f080b15b0 con 0x7f3f0813f710 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:40:01.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.934+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ee8004eb0 con 0x7f3f0813f710 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368107305 0 0) 0x7f3f080b1420 con 0x7f3f081430c0 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:40:01.935 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.935+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f080b15b0 con 0x7f3f081430c0 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826960984 0 0) 0x7f3f081db7d0 con 0x7f3f080a98f0 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:40:01.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.942+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f080b1420 con 0x7f3f080a98f0 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193869827 0 0) 0x7f3ee8004eb0 con 0x7f3f0813f710 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f081db7d0 con 0x7f3f0813f710 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3efc0019d0 con 0x7f3f0813f710 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.959 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.959+0000 7f3ef6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368302681 0 0) 0x7f3f080b15b0 con 0x7f3f081430c0 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ee8004eb0 con 0x7f3f081430c0 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f00001b90 con 0x7f3f081430c0 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.963+0000 7f3ef6ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.963 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:01.964 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:01.975 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328859242 0 0) 0x7f3ee8004eb0 con 0x7f3f081430c0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 >> v1:172.21.15.29:6790/0 conn(0x7f3f0813f710 legacy=0x7f3f080b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 >> v1:172.21.15.29:6789/0 conn(0x7f3f080a98f0 legacy=0x7f3f081daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 1 monclient: found mon.b 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080b1740 con 0x7f3f081430c0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3ef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 10 monclient: _renew_subs 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f081db1a0 con 0x7f3f081430c0 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:01.976 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 10 monclient: _renew_subs 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f080b08f0 con 0x7f3f081430c0 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.975+0000 7f3f0ca16640 1 librados: init done 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.976+0000 7f3f0ca16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.976+0000 7f3f0ca16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.976+0000 7f3f0ca16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:01.977 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:01.976+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f080627b0 con 0x7f3f081430c0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f00002040 con 0x7f3f081430c0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3f00004d00 con 0x7f3f081430c0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3f04d99640 4 set_mon_vals no callback set 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.279+0000 7f3ef6ffd640 10 monclient: dump: 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:02.283 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:02.284 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:02.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.280+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f3f0001acd0 con 0x7f3f081430c0 2024-03-25T19:40:02.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.281+0000 7f3f0659c640 10 monclient: get_auth_request con 0x7f3ee806dfb0 auth_method 0 2024-03-25T19:40:02.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.281+0000 7f3ef6ffd640 10 client.4501.objecter ms_handle_connect 0x7f3ee806dfb0 2024-03-25T19:40:02.285 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.283+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f3f000549c0 con 0x7f3f081430c0 2024-03-25T19:40:02.285 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.283+0000 7f3ef6ffd640 10 client.4501.objecter ms_dispatch 0x7f3f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:02.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.283+0000 7f3ef6ffd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:02.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.283+0000 7f3ef6ffd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:02.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.283+0000 7f3ef6ffd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T19:40:02.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.285+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f000504d0 con 0x7f3f081430c0 2024-03-25T19:40:02.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.285+0000 7f3ef6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.285+0000 7f3ef6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:02.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.528+0000 7f3f0ca16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.528+0000 7f3f0ca16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.528+0000 7f3f0ca16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:02.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.528+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f080abae0 con 0x7f3f081430c0 2024-03-25T19:40:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.571+0000 7f3ef6ffd640 1 -- 172.21.15.29:0/1687134144 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f3f00052400 con 0x7f3f081430c0 2024-03-25T19:40:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.571+0000 7f3ef6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.571+0000 7f3ef6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:02.571 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:02.571 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:02.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 10 librados: watch_flush enter 2024-03-25T19:40:02.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 10 librados: watch_flush exit 2024-03-25T19:40:02.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T19:40:02.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 10 client.4501.objecter successfully canceled tick 2024-03-25T19:40:02.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 >> v1:172.21.15.29:6800/2314944989 conn(0x7f3ee806dfb0 legacy=0x7f3ee8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 10 monclient: shutdown 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 >> v1:172.21.15.196:6789/0 conn(0x7f3f081430c0 legacy=0x7f3f081d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 shutdown_connections 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 >> 172.21.15.29:0/1687134144 conn(0x7f3f080b7ea0 msgr2=0x7f3f0813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 shutdown_connections 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 -- 172.21.15.29:0/1687134144 wait complete. 2024-03-25T19:40:02.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:02.574+0000 7f3f0ca16640 1 librados: shutdown 2024-03-25T19:40:02.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:09.390 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:09.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:09.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 10 monclient: monmap: 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:09.506812+0000 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:09.506812+0000 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:09.506 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 1 Processor -- start 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 1 -- start start 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.505+0000 7f9853ed6640 10 monclient: init 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f984c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f984c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f984c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0af940 con 0x7f984c0b4d60 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0b0a70 con 0x7f984c0b1730 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0b0d50 con 0x7f984c14be90 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.506+0000 7f9853ed6640 10 monclient(hunting): authenticate will time out at 6486.260485s 2024-03-25T19:40:09.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.510+0000 7f9851471640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f984c0b1730 0x7f984c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:60978/0 (socket says 172.21.15.29:60978) 2024-03-25T19:40:09.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.510+0000 7f9851471640 1 -- 172.21.15.29:0/274030677 learned_addr learned my addr 172.21.15.29:0/274030677 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.520+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458811610 0 0) 0x7f984c0b0a70 con 0x7f984c0b1730 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.521+0000 7f9850c70640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.521+0000 7f9850c70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.521+0000 7f9850c70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.521+0000 7f9850c70640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:40:09.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.521+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838003520 con 0x7f984c0b1730 2024-03-25T19:40:09.536 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 930295664 0 0) 0x7f9838003520 con 0x7f984c0b1730 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 >> v1:172.21.15.29:6790/0 conn(0x7f984c14be90 legacy=0x7f984c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 >> v1:172.21.15.29:6789/0 conn(0x7f984c0b4d60 legacy=0x7f984c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 monclient: found mon.noname-c 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0b1030 con 0x7f984c0b1730 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9853ed6640 5 monclient: authenticate success, global_id 4504 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9853ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:09.537 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f983c001690 con 0x7f984c0b1730 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9850c70640 10 monclient: dump: 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.538 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.535+0000 7f9853ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f983c001920 con 0x7f984c0b1730 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 1 -- 172.21.15.29:0/274030677 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f983c001cb0 con 0x7f984c0b1730 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9853ed6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9853ed6640 4 set_mon_vals no callback set 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.547+0000 7f9850c70640 10 monclient: dump: 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:09.548 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:09.549 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.548+0000 7f9853ed6640 10 monclient: shutdown 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.548+0000 7f9853ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.548+0000 7f9853ed6640 1 -- 172.21.15.29:0/274030677 >> v1:172.21.15.196:6789/0 conn(0x7f984c0b1730 legacy=0x7f984c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.548+0000 7f9853ed6640 1 -- 172.21.15.29:0/274030677 shutdown_connections 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.549+0000 7f9853ed6640 1 -- 172.21.15.29:0/274030677 >> 172.21.15.29:0/274030677 conn(0x7f984c0617a0 msgr2=0x7f984c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.549+0000 7f9853ed6640 1 -- 172.21.15.29:0/274030677 shutdown_connections 2024-03-25T19:40:09.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.549+0000 7f9853ed6640 1 -- 172.21.15.29:0/274030677 wait complete. 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 10 monclient: monmap: 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.551 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 librados: starting msgr at 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 librados: starting objecter 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 Processor -- start 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 -- start start 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 librados: setting wanted keys 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 1 librados: calling monclient init 2024-03-25T19:40:09.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.550+0000 7f9853ed6640 10 monclient: init 2024-03-25T19:40:09.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:09.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f984c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:09.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f984c0b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:09.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f984c14be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:09.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0aab40 con 0x7f984c14be90 2024-03-25T19:40:09.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0aae20 con 0x7f984c0b1730 2024-03-25T19:40:09.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c0a9ac0 con 0x7f984c0b4d60 2024-03-25T19:40:09.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.554 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.551+0000 7f9853ed6640 10 monclient(hunting): authenticate will time out at 6486.305810s 2024-03-25T19:40:09.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.567+0000 7f9851c72640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f984c0b1730 0x7f984c137f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:32798/0 (socket says 172.21.15.29:32798) 2024-03-25T19:40:09.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.567+0000 7f9851c72640 1 -- 172.21.15.29:0/2497607010 learned_addr learned my addr 172.21.15.29:0/2497607010 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:09.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.572+0000 7f98427fc640 10 client.?.objecter ms_handle_connect 0x7f984c0b1730 2024-03-25T19:40:09.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.572+0000 7f98427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.578+0000 7f98427fc640 10 client.?.objecter ms_handle_connect 0x7f984c14be90 2024-03-25T19:40:09.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.578+0000 7f98427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.579+0000 7f98427fc640 10 client.?.objecter ms_handle_connect 0x7f984c0b4d60 2024-03-25T19:40:09.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.579+0000 7f98427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303403899 0 0) 0x7f984c0aae20 con 0x7f984c0b1730 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:40:09.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.581+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9830004eb0 con 0x7f984c0b1730 2024-03-25T19:40:09.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.590+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009548902 0 0) 0x7f9830004eb0 con 0x7f984c0b1730 2024-03-25T19:40:09.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.590+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f984c0aae20 con 0x7f984c0b1730 2024-03-25T19:40:09.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983027471 0 0) 0x7f984c0aab40 con 0x7f984c14be90 2024-03-25T19:40:09.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:40:09.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.597+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9830004eb0 con 0x7f984c14be90 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121328119 0 0) 0x7f984c0a9ac0 con 0x7f984c0b4d60 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:40:09.601 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.600+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984c0aab40 con 0x7f984c0b4d60 2024-03-25T19:40:09.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9848004180 con 0x7f984c0b1730 2024-03-25T19:40:09.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:09.609 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient(hunting): dump: 2024-03-25T19:40:09.609 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 751502677 0 0) 0x7f984c0aae20 con 0x7f984c0b1730 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 >> v1:172.21.15.29:6790/0 conn(0x7f984c0b4d60 legacy=0x7f984c1d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 >> v1:172.21.15.29:6789/0 conn(0x7f984c14be90 legacy=0x7f984c1d8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 monclient: found mon.b 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0a9c50 con 0x7f984c0b1730 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f98427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 10 monclient: _renew_subs 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984c143310 con 0x7f984c0b1730 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 10 monclient: _renew_subs 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c140560 con 0x7f984c0b1730 2024-03-25T19:40:09.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.608+0000 7f9853ed6640 1 librados: init done 2024-03-25T19:40:09.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.609+0000 7f9853ed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.609+0000 7f9853ed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.609+0000 7f9853ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:09.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.609+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c0a9710 con 0x7f984c0b1730 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98480042e0 con 0x7f984c0b1730 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9848001320 con 0x7f984c0b1730 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f98427fc640 10 monclient: dump: 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:09.890 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.888+0000 7f9843fff640 4 set_mon_vals no callback set 2024-03-25T19:40:09.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.889+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9848001540 con 0x7f984c0b1730 2024-03-25T19:40:09.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.890+0000 7f9851471640 10 monclient: get_auth_request con 0x7f983006de80 auth_method 0 2024-03-25T19:40:09.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.890+0000 7f98427fc640 10 client.4507.objecter ms_handle_connect 0x7f983006de80 2024-03-25T19:40:09.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.894+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f98480547f0 con 0x7f984c0b1730 2024-03-25T19:40:09.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.894+0000 7f98427fc640 10 client.4507.objecter ms_dispatch 0x7f984c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:09.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.894+0000 7f98427fc640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:09.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.894+0000 7f98427fc640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:09.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.895+0000 7f98427fc640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T19:40:09.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.895+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9848050a90 con 0x7f984c0b1730 2024-03-25T19:40:09.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.895+0000 7f98427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:09.895+0000 7f98427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:10.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.130+0000 7f9853ed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.130+0000 7f9853ed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.130+0000 7f9853ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:10.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.130+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f984c0627b0 con 0x7f984c0b1730 2024-03-25T19:40:10.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.249+0000 7f98427fc640 1 -- 172.21.15.29:0/2497607010 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f9848050c30 con 0x7f984c0b1730 2024-03-25T19:40:10.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.249+0000 7f98427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.249+0000 7f98427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:10.250 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:10.250 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 10 librados: watch_flush enter 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 10 librados: watch_flush exit 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 10 client.4507.objecter successfully canceled tick 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 >> v1:172.21.15.29:6800/2314944989 conn(0x7f983006de80 legacy=0x7f9830070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 10 monclient: shutdown 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 >> v1:172.21.15.196:6789/0 conn(0x7f984c0b1730 legacy=0x7f984c137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.252 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 shutdown_connections 2024-03-25T19:40:10.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 >> 172.21.15.29:0/2497607010 conn(0x7f984c0617a0 msgr2=0x7f984c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:10.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 shutdown_connections 2024-03-25T19:40:10.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 -- 172.21.15.29:0/2497607010 wait complete. 2024-03-25T19:40:10.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:10.251+0000 7f9853ed6640 1 librados: shutdown 2024-03-25T19:40:10.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:10.532 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:40.696 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.547 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:40.711 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.569 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:40.733 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.588 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:40.751 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.598 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:40.760 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.616 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:40.779 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.622 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:40.786 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.626 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:06:40.788 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:10.652 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:40.816 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:17.068 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:17.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.183+0000 7ff1fb59e640 10 monclient: get_monmap_and_config 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.183+0000 7ff1fb59e640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.183+0000 7ff1fb59e640 10 monclient: monmap: 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:17.185500+0000 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:17.185500+0000 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.184+0000 7ff1fb59e640 1 Processor -- start 2024-03-25T19:40:17.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.184+0000 7ff1fb59e640 1 -- start start 2024-03-25T19:40:17.186 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.184+0000 7ff1fb59e640 10 monclient: init 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7ff1fc0b48e0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff1fc13fc60 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff1fc0b3f90 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0a9d10 con 0x7ff1fc13fc60 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0ab000 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0a9680 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.185+0000 7ff1fb59e640 10 monclient(hunting): authenticate will time out at 6493.939380s 2024-03-25T19:40:17.196 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.195+0000 7ff1fad9d640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7ff1fc0b3f90 0x7ff1fc14dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:57772/0 (socket says 172.21.15.29:57772) 2024-03-25T19:40:17.196 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.195+0000 7ff1fad9d640 1 -- 172.21.15.29:0/1982809478 learned_addr learned my addr 172.21.15.29:0/1982809478 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938538567 0 0) 0x7ff1fc0a9680 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:40:17.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.212+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4003520 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106318936 0 0) 0x7ff1fc0a9d10 con 0x7ff1fc13fc60 2024-03-25T19:40:17.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:40:17.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.219+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc0a9680 con 0x7ff1fc13fc60 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557146854 0 0) 0x7ff1e4003520 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 >> v1:172.21.15.196:6789/0 conn(0x7ff1fc0b48e0 legacy=0x7ff1fc13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 >> v1:172.21.15.29:6789/0 conn(0x7ff1fc13fc60 legacy=0x7ff1fc145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 monclient: found mon.noname-b 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0a9810 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.226 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f4001a10 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1f959a640 10 monclient: dump: 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.227 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1fb59e640 5 monclient: authenticate success, global_id 4511 2024-03-25T19:40:17.228 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.224+0000 7ff1fb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:17.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f4001b70 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 1 -- 172.21.15.29:0/1982809478 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f4001f80 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1f959a640 10 monclient: dump: 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1fb59e640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:17.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1fb59e640 4 set_mon_vals no callback set 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1fb59e640 10 monclient: shutdown 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.250+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1982809478 >> v1:172.21.15.29:6790/0 conn(0x7ff1fc0b3f90 legacy=0x7ff1fc14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.251+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1982809478 shutdown_connections 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.251+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1982809478 >> 172.21.15.29:0/1982809478 conn(0x7ff1fc0615c0 msgr2=0x7ff1fc0619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.251+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1982809478 shutdown_connections 2024-03-25T19:40:17.253 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.251+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1982809478 wait complete. 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 10 monclient: monmap: 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 librados: starting msgr at 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 librados: starting objecter 2024-03-25T19:40:17.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 Processor -- start 2024-03-25T19:40:17.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 -- start start 2024-03-25T19:40:17.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 librados: setting wanted keys 2024-03-25T19:40:17.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 1 librados: calling monclient init 2024-03-25T19:40:17.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.252+0000 7ff1fb59e640 10 monclient: init 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff1fc0b3f90 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff1fc0b48e0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7ff1fc13fc60 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0ae150 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc140dc0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc140f50 con 0x7ff1fc13fc60 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.256 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.253+0000 7ff1fb59e640 10 monclient(hunting): authenticate will time out at 6494.007445s 2024-03-25T19:40:17.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.258+0000 7ff1f9d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff1fc0b48e0 0x7ff1fc1db730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:60120/0 (socket says 172.21.15.29:60120) 2024-03-25T19:40:17.259 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.258+0000 7ff1f9d9b640 1 -- 172.21.15.29:0/1974616428 learned_addr learned my addr 172.21.15.29:0/1974616428 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:17.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.267+0000 7ff1eaffd640 10 client.?.objecter ms_handle_connect 0x7ff1fc0b48e0 2024-03-25T19:40:17.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.267+0000 7ff1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69598000 0 0) 0x7ff1fc140dc0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:40:17.274 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.273+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.275 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.274+0000 7ff1eaffd640 10 client.?.objecter ms_handle_connect 0x7ff1fc13fc60 2024-03-25T19:40:17.275 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.274+0000 7ff1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.285+0000 7ff1eaffd640 10 client.?.objecter ms_handle_connect 0x7ff1fc0b3f90 2024-03-25T19:40:17.286 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.285+0000 7ff1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738987994 0 0) 0x7ff1fc0a9810 con 0x7ff1fc13fc60 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:40:17.289 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.288+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc140dc0 con 0x7ff1fc13fc60 2024-03-25T19:40:17.292 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830514095 0 0) 0x7ff1fc0ae150 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.292 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.293 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.293 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.293 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:40:17.293 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.292+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc0a9810 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4021165570 0 0) 0x7ff1fc140dc0 con 0x7ff1fc13fc60 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1fc0ae150 con 0x7ff1fc13fc60 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f4001ce0 con 0x7ff1fc13fc60 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.298 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.297+0000 7ff1eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533484070 0 0) 0x7ff1dc004eb0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1fc140dc0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f0001930 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.298+0000 7ff1eaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.300 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.301 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.301 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.301 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209226453 0 0) 0x7ff1fc0a9810 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1ec001860 con 0x7ff1fc0b3f90 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.312+0000 7ff1eaffd640 10 monclient(hunting): dump: 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.313 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220054826 0 0) 0x7ff1fc140dc0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 >> v1:172.21.15.29:6790/0 conn(0x7ff1fc13fc60 legacy=0x7ff1fc137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 >> v1:172.21.15.29:6789/0 conn(0x7ff1fc0b3f90 legacy=0x7ff1fc0ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 1 monclient: found mon.b 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0acf40 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 10 monclient: _renew_subs 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc1dbe40 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 10 monclient: _renew_subs 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc0abed0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.315+0000 7ff1fb59e640 1 librados: init done 2024-03-25T19:40:17.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.316+0000 7ff1fb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.316+0000 7ff1fb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.316+0000 7ff1fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:17.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.316+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc0b2070 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f0003610 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff1f00039e0 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1eaffd640 10 monclient: dump: 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.590+0000 7ff1f8d99640 4 set_mon_vals no callback set 2024-03-25T19:40:17.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.591+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ff1f0003c00 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.592+0000 7ff1fa59c640 10 monclient: get_auth_request con 0x7ff1dc06e160 auth_method 0 2024-03-25T19:40:17.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.592+0000 7ff1eaffd640 10 client.4513.objecter ms_handle_connect 0x7ff1dc06e160 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.597+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ff1f0054740 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.597+0000 7ff1eaffd640 10 client.4513.objecter ms_dispatch 0x7ff1fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.597+0000 7ff1eaffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.597+0000 7ff1eaffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.597+0000 7ff1eaffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.598+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1f0050250 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.598+0000 7ff1eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.598+0000 7ff1eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:17.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.845+0000 7ff1fb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:17.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.845+0000 7ff1fb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:17.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.845+0000 7ff1fb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:17.846 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.845+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1fc13e800 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.962+0000 7ff1eaffd640 1 -- 172.21.15.29:0/1974616428 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ff1f0052180 con 0x7ff1fc0b48e0 2024-03-25T19:40:17.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.962+0000 7ff1eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:17.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.962+0000 7ff1eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:17.964 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:17.964 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:17.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 10 librados: watch_flush enter 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 10 librados: watch_flush exit 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 >> v1:172.21.15.29:6800/2314944989 conn(0x7ff1dc06e160 legacy=0x7ff1dc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 10 monclient: shutdown 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 >> v1:172.21.15.196:6789/0 conn(0x7ff1fc0b48e0 legacy=0x7ff1fc1db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 shutdown_connections 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 >> 172.21.15.29:0/1974616428 conn(0x7ff1fc0615c0 msgr2=0x7ff1fc13c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 shutdown_connections 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.965+0000 7ff1fb59e640 1 -- 172.21.15.29:0/1974616428 wait complete. 2024-03-25T19:40:17.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:17.966+0000 7ff1fb59e640 1 librados: shutdown 2024-03-25T19:40:17.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:24.785 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.902+0000 7fbf6fe8f640 10 monclient: get_monmap_and_config 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.902+0000 7fbf6fe8f640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.902+0000 7fbf6fe8f640 10 monclient: monmap: 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:24.902 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:24.903170+0000 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:24.903170+0000 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:24.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 1 Processor -- start 2024-03-25T19:40:24.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 1 -- start start 2024-03-25T19:40:24.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient: init 2024-03-25T19:40:24.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.904 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-b con 0x7fbf680a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-a con 0x7fbf6813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-c con 0x7fbf681430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf68142520 con 0x7fbf6813f710 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680b28a0 con 0x7fbf681430c0 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680b2a30 con 0x7fbf680a98f0 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.903+0000 7fbf6fe8f640 10 monclient(hunting): authenticate will time out at 6501.656821s 2024-03-25T19:40:24.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.915+0000 7fbf6d42a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf6813f710 0x7fbf681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57006/0 (socket says 172.21.15.29:57006) 2024-03-25T19:40:24.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.915+0000 7fbf6d42a640 1 -- 172.21.15.29:0/1509585018 learned_addr learned my addr 172.21.15.29:0/1509585018 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:24.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737649983 0 0) 0x7fbf680b2a30 con 0x7fbf680a98f0 2024-03-25T19:40:24.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:40:24.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.932+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003520 con 0x7fbf680a98f0 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.941+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283007598 0 0) 0x7fbf68142520 con 0x7fbf6813f710 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.941+0000 7fbf6cc29640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.941+0000 7fbf6cc29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.941+0000 7fbf6cc29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.941+0000 7fbf6cc29640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:40:24.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.942+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf680b2a30 con 0x7fbf6813f710 2024-03-25T19:40:24.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4140951466 0 0) 0x7fbf680b2a30 con 0x7fbf6813f710 2024-03-25T19:40:24.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 >> v1:172.21.15.29:6790/0 conn(0x7fbf680a98f0 legacy=0x7fbf680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 >> v1:172.21.15.196:6789/0 conn(0x7fbf681430c0 legacy=0x7fbf6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 monclient: found mon.noname-a 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680b44c0 con 0x7fbf6813f710 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf58001670 con 0x7fbf6813f710 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6cc29640 10 monclient: dump: 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:24.950 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6fe8f640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:40:24.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.949+0000 7fbf6fe8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf580017d0 con 0x7fbf6813f710 2024-03-25T19:40:24.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 1 -- 172.21.15.29:0/1509585018 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf58001c20 con 0x7fbf6813f710 2024-03-25T19:40:24.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6cc29640 10 monclient: dump: 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6fe8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:24.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.960+0000 7fbf6fe8f640 4 set_mon_vals no callback set 2024-03-25T19:40:24.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 10 monclient: shutdown 2024-03-25T19:40:24.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.961 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/1509585018 >> v1:172.21.15.29:6789/0 conn(0x7fbf6813f710 legacy=0x7fbf681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.962 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/1509585018 shutdown_connections 2024-03-25T19:40:24.962 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/1509585018 >> 172.21.15.29:0/1509585018 conn(0x7fbf680b7ea0 msgr2=0x7fbf680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.962 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/1509585018 shutdown_connections 2024-03-25T19:40:24.962 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.961+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/1509585018 wait complete. 2024-03-25T19:40:24.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.962+0000 7fbf6fe8f640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.962+0000 7fbf6fe8f640 10 monclient: monmap: 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 librados: starting msgr at 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 librados: starting objecter 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 Processor -- start 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 -- start start 2024-03-25T19:40:24.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 librados: setting wanted keys 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 librados: calling monclient init 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient: init 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-b con 0x7fbf680a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-c con 0x7fbf6813f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient(hunting): picked mon.noname-a con 0x7fbf681430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680b7a60 con 0x7fbf681430c0 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf681dc070 con 0x7fbf680a98f0 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6813b110 con 0x7fbf6813f710 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.963+0000 7fbf6fe8f640 10 monclient(hunting): authenticate will time out at 6501.716848s 2024-03-25T19:40:24.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.985+0000 7fbf6dc2b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbf680a98f0 0x7fbf68139db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36592/0 (socket says 172.21.15.29:36592) 2024-03-25T19:40:24.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.985+0000 7fbf6dc2b640 1 -- 172.21.15.29:0/752976093 learned_addr learned my addr 172.21.15.29:0/752976093 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:24.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.986+0000 7fbf5e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf6813f710 2024-03-25T19:40:24.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.986+0000 7fbf5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.987+0000 7fbf5e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf680a98f0 2024-03-25T19:40:24.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.987+0000 7fbf5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.991 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.992+0000 7fbf5e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf681430c0 2024-03-25T19:40:24.991 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.992+0000 7fbf5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982463080 0 0) 0x7fbf6813b110 con 0x7fbf6813f710 2024-03-25T19:40:24.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:40:24.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:24.998+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf4c004eb0 con 0x7fbf6813f710 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964829831 0 0) 0x7fbf680b7a60 con 0x7fbf681430c0 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:40:25.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.000+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6813b110 con 0x7fbf681430c0 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153064812 0 0) 0x7fbf681dc070 con 0x7fbf680a98f0 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:40:25.001 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.001+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf680b7a60 con 0x7fbf680a98f0 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3286540028 0 0) 0x7fbf680b7a60 con 0x7fbf680a98f0 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf681dc070 con 0x7fbf680a98f0 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf64001950 con 0x7fbf680a98f0 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:25.017 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667457674 0 0) 0x7fbf4c004eb0 con 0x7fbf6813f710 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf680b7a60 con 0x7fbf6813f710 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf58001ba0 con 0x7fbf6813f710 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.018 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.017+0000 7fbf5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:25.019 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 215615525 0 0) 0x7fbf6813b110 con 0x7fbf681430c0 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf4c004eb0 con 0x7fbf681430c0 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf60001b00 con 0x7fbf681430c0 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.020+0000 7fbf5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.020 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:25.021 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:25.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 12961921 0 0) 0x7fbf4c004eb0 con 0x7fbf681430c0 2024-03-25T19:40:25.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 >> v1:172.21.15.29:6790/0 conn(0x7fbf6813f710 legacy=0x7fbf681db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.033 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 >> v1:172.21.15.196:6789/0 conn(0x7fbf680a98f0 legacy=0x7fbf68139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 1 monclient: found mon.a 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6813b3f0 con 0x7fbf681430c0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf6fe8f640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf6fe8f640 10 monclient: _renew_subs 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf6fe8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6813b6d0 con 0x7fbf681430c0 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.033+0000 7fbf6fe8f640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:25.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.034+0000 7fbf6fe8f640 10 monclient: _renew_subs 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.034+0000 7fbf6fe8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.034+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6813a4c0 con 0x7fbf681430c0 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.034+0000 7fbf6fe8f640 1 librados: init done 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.035+0000 7fbf6fe8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.035+0000 7fbf6fe8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.035+0000 7fbf6fe8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:25.036 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.035+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf680b3ef0 con 0x7fbf681430c0 2024-03-25T19:40:25.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf60001fd0 con 0x7fbf681430c0 2024-03-25T19:40:25.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbf60003b10 con 0x7fbf681430c0 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5e7fc640 10 monclient: dump: 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:25.348 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:25.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.348+0000 7fbf5ffff640 4 set_mon_vals no callback set 2024-03-25T19:40:25.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.349+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbf60019ae0 con 0x7fbf681430c0 2024-03-25T19:40:25.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.349+0000 7fbf6dc2b640 10 monclient: get_auth_request con 0x7fbf4c06e110 auth_method 0 2024-03-25T19:40:25.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.349+0000 7fbf5e7fc640 10 client.4518.objecter ms_handle_connect 0x7fbf4c06e110 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fbf600537a0 con 0x7fbf681430c0 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 10 client.4518.objecter ms_dispatch 0x7fbf68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf6004f0e0 con 0x7fbf681430c0 2024-03-25T19:40:25.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.357+0000 7fbf5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.592+0000 7fbf6fe8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.592+0000 7fbf6fe8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.592+0000 7fbf6fe8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:25.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.592+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf680aa310 con 0x7fbf681430c0 2024-03-25T19:40:25.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.652+0000 7fbf5e7fc640 1 -- 172.21.15.29:0/752976093 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fbf600511e0 con 0x7fbf681430c0 2024-03-25T19:40:25.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.652+0000 7fbf5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.652+0000 7fbf5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:25.652 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:25.652 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 10 librados: watch_flush enter 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 10 librados: watch_flush exit 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 10 client.4518.objecter successfully canceled tick 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbf4c06e110 legacy=0x7fbf4c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 10 monclient: shutdown 2024-03-25T19:40:25.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 >> v1:172.21.15.29:6789/0 conn(0x7fbf681430c0 legacy=0x7fbf681db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 shutdown_connections 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 >> 172.21.15.29:0/752976093 conn(0x7fbf680b7ea0 msgr2=0x7fbf68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 shutdown_connections 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 -- 172.21.15.29:0/752976093 wait complete. 2024-03-25T19:40:25.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:25.655+0000 7fbf6fe8f640 1 librados: shutdown 2024-03-25T19:40:25.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:30.592 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:00.755 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.596 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:00.759 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.622 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:00.786 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.653 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:00.816 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.657 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:00.820 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.667 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:00.830 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.676 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:00.839 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.681 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:00.843 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:30.715 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:00.878 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:32.471 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:32.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.587+0000 7fea45a72640 10 monclient: get_monmap_and_config 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.587+0000 7fea45a72640 10 monclient: build_initial_monmap 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.587+0000 7fea45a72640 10 monclient: monmap: 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:32.588119+0000 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:32.588119+0000 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:32.587 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 1 Processor -- start 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 1 -- start start 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 10 monclient: init 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-b con 0x7fea400a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.588+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-c con 0x7fea4013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-a con 0x7fea401430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40142520 con 0x7fea401430c0 2024-03-25T19:40:32.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400b28a0 con 0x7fea4013f710 2024-03-25T19:40:32.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400b2a30 con 0x7fea400a98f0 2024-03-25T19:40:32.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.589+0000 7fea45a72640 10 monclient(hunting): authenticate will time out at 6509.341943s 2024-03-25T19:40:32.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.591+0000 7fea3effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fea4013f710 0x7fea401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:43640/0 (socket says 172.21.15.29:43640) 2024-03-25T19:40:32.590 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.591+0000 7fea3effd640 1 -- 172.21.15.29:0/2246568358 learned_addr learned my addr 172.21.15.29:0/2246568358 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738339775 0 0) 0x7fea400b28a0 con 0x7fea4013f710 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:40:32.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.599+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28003520 con 0x7fea4013f710 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959058260 0 0) 0x7fea40142520 con 0x7fea401430c0 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea400b28a0 con 0x7fea401430c0 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 845785082 0 0) 0x7fea28003520 con 0x7fea4013f710 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 >> v1:172.21.15.29:6790/0 conn(0x7fea400a98f0 legacy=0x7fea400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 >> v1:172.21.15.29:6789/0 conn(0x7fea401430c0 legacy=0x7fea4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:32.604 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400b44c0 con 0x7fea4013f710 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fea34001690 con 0x7fea4013f710 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea3e7fc640 10 monclient: dump: 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:32.605 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:32.606 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:32.606 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:32.606 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea45a72640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:40:32.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.604+0000 7fea45a72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea340017f0 con 0x7fea4013f710 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea3e7fc640 1 -- 172.21.15.29:0/2246568358 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fea34001c00 con 0x7fea4013f710 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea45a72640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea45a72640 4 set_mon_vals no callback set 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.607+0000 7fea3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea3e7fc640 10 monclient: dump: 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:32.608 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 10 monclient: shutdown 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 1 -- 172.21.15.29:0/2246568358 >> v1:172.21.15.196:6789/0 conn(0x7fea4013f710 legacy=0x7fea401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 1 -- 172.21.15.29:0/2246568358 shutdown_connections 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 1 -- 172.21.15.29:0/2246568358 >> 172.21.15.29:0/2246568358 conn(0x7fea400b7ea0 msgr2=0x7fea400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 1 -- 172.21.15.29:0/2246568358 shutdown_connections 2024-03-25T19:40:32.610 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.608+0000 7fea45a72640 1 -- 172.21.15.29:0/2246568358 wait complete. 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.609+0000 7fea45a72640 10 monclient: build_initial_monmap 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.609+0000 7fea45a72640 10 monclient: monmap: 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 librados: starting msgr at 2024-03-25T19:40:32.611 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 librados: starting objecter 2024-03-25T19:40:32.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 Processor -- start 2024-03-25T19:40:32.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 -- start start 2024-03-25T19:40:32.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 librados: setting wanted keys 2024-03-25T19:40:32.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 1 librados: calling monclient init 2024-03-25T19:40:32.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.610+0000 7fea45a72640 10 monclient: init 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-b con 0x7fea400a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-c con 0x7fea4013f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient(hunting): picked mon.noname-a con 0x7fea401430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40138ac0 con 0x7fea401430c0 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40137af0 con 0x7fea400a98f0 2024-03-25T19:40:32.613 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40136740 con 0x7fea4013f710 2024-03-25T19:40:32.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.611+0000 7fea45a72640 10 monclient(hunting): authenticate will time out at 6509.364191s 2024-03-25T19:40:32.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.612+0000 7fea3f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fea400a98f0 0x7fea40139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:43654/0 (socket says 172.21.15.29:43654) 2024-03-25T19:40:32.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.612+0000 7fea3f7fe640 1 -- 172.21.15.29:0/1434729993 learned_addr learned my addr 172.21.15.29:0/1434729993 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:32.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.617+0000 7fea27fff640 10 client.?.objecter ms_handle_connect 0x7fea400a98f0 2024-03-25T19:40:32.616 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.617+0000 7fea27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.624+0000 7fea27fff640 10 client.?.objecter ms_handle_connect 0x7fea401430c0 2024-03-25T19:40:32.624 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.624+0000 7fea27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154773311 0 0) 0x7fea40137af0 con 0x7fea400a98f0 2024-03-25T19:40:32.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.628 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:40:32.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.629+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c004eb0 con 0x7fea400a98f0 2024-03-25T19:40:32.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.637+0000 7fea27fff640 10 client.?.objecter ms_handle_connect 0x7fea4013f710 2024-03-25T19:40:32.637 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.637+0000 7fea27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315845979 0 0) 0x7fea40138ac0 con 0x7fea401430c0 2024-03-25T19:40:32.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.639 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:40:32.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.640+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40137af0 con 0x7fea401430c0 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596129977 0 0) 0x7fea2c004eb0 con 0x7fea400a98f0 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea40138ac0 con 0x7fea400a98f0 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fea30004080 con 0x7fea400a98f0 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 10 monclient(hunting): dump: 2024-03-25T19:40:32.647 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.647+0000 7fea27fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093525108 0 0) 0x7fea40136740 con 0x7fea4013f710 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:40:32.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.648+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c004eb0 con 0x7fea4013f710 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.654+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 67743866 0 0) 0x7fea40138ac0 con 0x7fea400a98f0 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 >> v1:172.21.15.29:6790/0 conn(0x7fea4013f710 legacy=0x7fea40134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 >> v1:172.21.15.29:6789/0 conn(0x7fea401430c0 legacy=0x7fea400b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 1 monclient: found mon.b 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4013b0b0 con 0x7fea400a98f0 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 10 monclient: _renew_subs 2024-03-25T19:40:32.655 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4013b380 con 0x7fea400a98f0 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 10 monclient: _renew_subs 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea401376c0 con 0x7fea400a98f0 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.655+0000 7fea45a72640 1 librados: init done 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.656+0000 7fea45a72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.656+0000 7fea45a72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:32.656 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.656+0000 7fea45a72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:32.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.656+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea401392c0 con 0x7fea400a98f0 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea30000ed0 con 0x7fea400a98f0 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fea30001220 con 0x7fea400a98f0 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea27fff640 10 monclient: dump: 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:32.676 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:32.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.676+0000 7fea3dffb640 4 set_mon_vals no callback set 2024-03-25T19:40:32.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.677+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fea30001400 con 0x7fea400a98f0 2024-03-25T19:40:32.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.678+0000 7fea3effd640 10 monclient: get_auth_request con 0x7fea2c06ddb0 auth_method 0 2024-03-25T19:40:32.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.678+0000 7fea27fff640 10 client.4525.objecter ms_handle_connect 0x7fea2c06ddb0 2024-03-25T19:40:32.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.679+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fea30054820 con 0x7fea400a98f0 2024-03-25T19:40:32.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.679+0000 7fea27fff640 10 client.4525.objecter ms_dispatch 0x7fea40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:32.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.679+0000 7fea27fff640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:32.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.679+0000 7fea27fff640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:32.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.679+0000 7fea27fff640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T19:40:32.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.956+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea30050330 con 0x7fea400a98f0 2024-03-25T19:40:32.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.956+0000 7fea27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:32.956 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:32.956+0000 7fea27fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:33.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.216+0000 7fea45a72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.216+0000 7fea45a72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.216+0000 7fea45a72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:40:33.216 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.216+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea400b3ef0 con 0x7fea400a98f0 2024-03-25T19:40:33.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.239+0000 7fea27fff640 1 -- 172.21.15.29:0/1434729993 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fea30052260 con 0x7fea400a98f0 2024-03-25T19:40:33.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.239+0000 7fea27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.239+0000 7fea27fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:33.239 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:33.239 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 10 librados: watch_flush enter 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 10 librados: watch_flush exit 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 10 client.4525.objecter successfully canceled tick 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 >> v1:172.21.15.29:6800/2314944989 conn(0x7fea2c06ddb0 legacy=0x7fea2c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 10 monclient: shutdown 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 >> v1:172.21.15.196:6789/0 conn(0x7fea400a98f0 legacy=0x7fea40139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:33.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 shutdown_connections 2024-03-25T19:40:33.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 >> 172.21.15.29:0/1434729993 conn(0x7fea400b7ea0 msgr2=0x7fea4013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:33.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 shutdown_connections 2024-03-25T19:40:33.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 -- 172.21.15.29:0/1434729993 wait complete. 2024-03-25T19:40:33.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:33.242+0000 7fea45a72640 1 librados: shutdown 2024-03-25T19:40:33.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:40.060 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.179+0000 7fb366f4d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.179+0000 7fb366f4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.179+0000 7fb366f4d640 10 monclient: monmap: 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:40.180237+0000 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:40.180237+0000 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:40.179 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:40.180 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.180+0000 7fb366f4d640 1 Processor -- start 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.180+0000 7fb366f4d640 1 -- start start 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.180+0000 7fb366f4d640 10 monclient: init 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3600a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fb36013f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3601430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb360142520 con 0x7fb36013f710 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3600b28a0 con 0x7fb3600a98f0 2024-03-25T19:40:40.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3600b2a30 con 0x7fb3601430c0 2024-03-25T19:40:40.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:40.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.181+0000 7fb366f4d640 10 monclient(hunting): authenticate will time out at 6516.934312s 2024-03-25T19:40:40.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.186+0000 7fb3654ea640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fb3601430c0 0x7fb36014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:50744/0 (socket says 172.21.15.29:50744) 2024-03-25T19:40:40.185 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.186+0000 7fb3654ea640 1 -- 172.21.15.29:0/524773359 learned_addr learned my addr 172.21.15.29:0/524773359 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:40.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180119654 0 0) 0x7fb3600b2a30 con 0x7fb3601430c0 2024-03-25T19:40:40.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:40:40.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.209+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb350003520 con 0x7fb3601430c0 2024-03-25T19:40:40.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572548200 0 0) 0x7fb3600b28a0 con 0x7fb3600a98f0 2024-03-25T19:40:40.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:40:40.213 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.213+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3600b2a30 con 0x7fb3600a98f0 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828310600 0 0) 0x7fb360142520 con 0x7fb36013f710 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:40:40.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.215+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3600b28a0 con 0x7fb36013f710 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325000265 0 0) 0x7fb3600b2a30 con 0x7fb3600a98f0 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 >> v1:172.21.15.29:6790/0 conn(0x7fb3601430c0 legacy=0x7fb36014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 >> v1:172.21.15.29:6789/0 conn(0x7fb36013f710 legacy=0x7fb3601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600b44c0 con 0x7fb3600a98f0 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:40.234 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb354001670 con 0x7fb3600a98f0 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb35f7fe640 10 monclient: dump: 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb366f4d640 5 monclient: authenticate success, global_id 4528 2024-03-25T19:40:40.235 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.234+0000 7fb366f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:40.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3540017d0 con 0x7fb3600a98f0 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 1 -- 172.21.15.29:0/524773359 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb354001be0 con 0x7fb3600a98f0 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb366f4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb366f4d640 4 set_mon_vals no callback set 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.264+0000 7fb35f7fe640 10 monclient: dump: 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:40.264 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:40.265 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:40.265 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:40.265 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 10 monclient: shutdown 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 1 -- 172.21.15.29:0/524773359 >> v1:172.21.15.196:6789/0 conn(0x7fb3600a98f0 legacy=0x7fb3600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 1 -- 172.21.15.29:0/524773359 shutdown_connections 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 1 -- 172.21.15.29:0/524773359 >> 172.21.15.29:0/524773359 conn(0x7fb3600b7ea0 msgr2=0x7fb3600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 1 -- 172.21.15.29:0/524773359 shutdown_connections 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.265+0000 7fb366f4d640 1 -- 172.21.15.29:0/524773359 wait complete. 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 10 monclient: monmap: 2024-03-25T19:40:40.266 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 librados: starting msgr at 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 librados: starting objecter 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 Processor -- start 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 -- start start 2024-03-25T19:40:40.267 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 librados: setting wanted keys 2024-03-25T19:40:40.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 1 librados: calling monclient init 2024-03-25T19:40:40.268 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.266+0000 7fb366f4d640 10 monclient: init 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3600a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fb36013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3601430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36013c190 con 0x7fb3601430c0 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3600b7cf0 con 0x7fb36013f710 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36013ad50 con 0x7fb3600a98f0 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:40.269 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.267+0000 7fb366f4d640 10 monclient(hunting): authenticate will time out at 6517.020697s 2024-03-25T19:40:40.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.279+0000 7fb364ce9640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fb3600a98f0 0x7fb36013ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:50826/0 (socket says 172.21.15.29:50826) 2024-03-25T19:40:40.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.279+0000 7fb364ce9640 1 -- 172.21.15.29:0/2686415111 learned_addr learned my addr 172.21.15.29:0/2686415111 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:40.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.280+0000 7fb35d7fa640 10 client.?.objecter ms_handle_connect 0x7fb36013f710 2024-03-25T19:40:40.280 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.280+0000 7fb35d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.283+0000 7fb35d7fa640 10 client.?.objecter ms_handle_connect 0x7fb3600a98f0 2024-03-25T19:40:40.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.283+0000 7fb35d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.284+0000 7fb35d7fa640 10 client.?.objecter ms_handle_connect 0x7fb3601430c0 2024-03-25T19:40:40.283 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.284+0000 7fb35d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.294 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691757148 0 0) 0x7fb3600b7cf0 con 0x7fb36013f710 2024-03-25T19:40:40.294 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.294 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.294 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.294 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:40:40.295 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.294+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344004eb0 con 0x7fb36013f710 2024-03-25T19:40:40.295 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511142722 0 0) 0x7fb350003520 con 0x7fb3601430c0 2024-03-25T19:40:40.295 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.295 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:40:40.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.296+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3600b7cf0 con 0x7fb3601430c0 2024-03-25T19:40:40.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310785112 0 0) 0x7fb36013ad50 con 0x7fb3600a98f0 2024-03-25T19:40:40.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:40:40.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.303+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb350003520 con 0x7fb3600a98f0 2024-03-25T19:40:40.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19271146 0 0) 0x7fb3600b7cf0 con 0x7fb3601430c0 2024-03-25T19:40:40.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb36013ad50 con 0x7fb3601430c0 2024-03-25T19:40:40.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb3580018d0 con 0x7fb3601430c0 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 10 monclient(hunting): dump: 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.312+0000 7fb35d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:40.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1501726733 0 0) 0x7fb36013ad50 con 0x7fb3601430c0 2024-03-25T19:40:40.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 >> v1:172.21.15.29:6790/0 conn(0x7fb3600a98f0 legacy=0x7fb36013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 >> v1:172.21.15.196:6789/0 conn(0x7fb36013f710 legacy=0x7fb3601df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.321 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 1 monclient: found mon.a 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36013afc0 con 0x7fb3601430c0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb35d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 10 monclient: _renew_subs 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36013b2a0 con 0x7fb3601430c0 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:40.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 10 monclient: _renew_subs 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36012fae0 con 0x7fb3601430c0 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.321+0000 7fb366f4d640 1 librados: init done 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.322+0000 7fb366f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.322+0000 7fb366f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.322+0000 7fb366f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:40.324 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.322+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3600b3ef0 con 0x7fb3601430c0 2024-03-25T19:40:40.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb358001d00 con 0x7fb3601430c0 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb358004b60 con 0x7fb3601430c0 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35d7fa640 10 monclient: dump: 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:40.641 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:40.642 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:40.642 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:40.642 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:40.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.641+0000 7fb35effd640 4 set_mon_vals no callback set 2024-03-25T19:40:40.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.642+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fb358004d40 con 0x7fb3601430c0 2024-03-25T19:40:40.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.642+0000 7fb364ce9640 10 monclient: get_auth_request con 0x7fb34406de20 auth_method 0 2024-03-25T19:40:40.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.643+0000 7fb35d7fa640 10 client.4530.objecter ms_handle_connect 0x7fb34406de20 2024-03-25T19:40:40.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.647+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fb358054810 con 0x7fb3601430c0 2024-03-25T19:40:40.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.647+0000 7fb35d7fa640 10 client.4530.objecter ms_dispatch 0x7fb360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:40.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.647+0000 7fb35d7fa640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:40.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.647+0000 7fb35d7fa640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:40.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.648+0000 7fb35d7fa640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-25T19:40:40.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.648+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb358050320 con 0x7fb3601430c0 2024-03-25T19:40:40.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.648+0000 7fb35d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.648+0000 7fb35d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.903+0000 7fb366f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.903+0000 7fb366f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.903+0000 7fb366f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.903+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3600627b0 con 0x7fb3601430c0 2024-03-25T19:40:40.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.955+0000 7fb35d7fa640 1 -- 172.21.15.29:0/2686415111 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fb358052250 con 0x7fb3601430c0 2024-03-25T19:40:40.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.955+0000 7fb35d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:40.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.955+0000 7fb35d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:40.955 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:40.956 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:40.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 10 librados: watch_flush enter 2024-03-25T19:40:40.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 10 librados: watch_flush exit 2024-03-25T19:40:40.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-25T19:40:40.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 10 client.4530.objecter successfully canceled tick 2024-03-25T19:40:40.958 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 >> v1:172.21.15.29:6800/2314944989 conn(0x7fb34406de20 legacy=0x7fb3440702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 10 monclient: shutdown 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 >> v1:172.21.15.29:6789/0 conn(0x7fb3601430c0 legacy=0x7fb3601e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 shutdown_connections 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 >> 172.21.15.29:0/2686415111 conn(0x7fb3600b7ea0 msgr2=0x7fb360074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 shutdown_connections 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 -- 172.21.15.29:0/2686415111 wait complete. 2024-03-25T19:40:40.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:40.958+0000 7fb366f4d640 1 librados: shutdown 2024-03-25T19:40:40.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:47.777 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:47.899 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.899+0000 7f318fa8d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.899+0000 7f318fa8d640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.899+0000 7f318fa8d640 10 monclient: monmap: 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:47.900734+0000 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:47.900734+0000 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:47.900 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:47.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.900+0000 7f318fa8d640 1 Processor -- start 2024-03-25T19:40:47.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.900+0000 7f318fa8d640 1 -- start start 2024-03-25T19:40:47.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.900+0000 7f318fa8d640 10 monclient: init 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f31880a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f318813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f31881430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188142520 con 0x7f31881430c0 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880b28a0 con 0x7f318813f710 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880b2a30 con 0x7f31880a98f0 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.902 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.901+0000 7f318fa8d640 10 monclient(hunting): authenticate will time out at 6524.654366s 2024-03-25T19:40:47.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.912+0000 7f318d829640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f31880a98f0 0x7f31880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:39058/0 (socket says 172.21.15.29:39058) 2024-03-25T19:40:47.912 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.912+0000 7f318d829640 1 -- 172.21.15.29:0/2359993723 learned_addr learned my addr 172.21.15.29:0/2359993723 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657623974 0 0) 0x7f31880b2a30 con 0x7f31880a98f0 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:40:47.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.929+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3174003520 con 0x7f31880a98f0 2024-03-25T19:40:47.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724854081 0 0) 0x7f3188142520 con 0x7f31881430c0 2024-03-25T19:40:47.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:40:47.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.938+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31880b2a30 con 0x7f31881430c0 2024-03-25T19:40:47.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134085078 0 0) 0x7f31880b28a0 con 0x7f318813f710 2024-03-25T19:40:47.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:40:47.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.940+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3188142520 con 0x7f318813f710 2024-03-25T19:40:47.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.951+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362011104 0 0) 0x7f3174003520 con 0x7f31880a98f0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.951+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 >> v1:172.21.15.196:6789/0 conn(0x7f318813f710 legacy=0x7f31881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.951+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 >> v1:172.21.15.29:6789/0 conn(0x7f31881430c0 legacy=0x7f318814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: _finish_hunting 0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 1 monclient: found mon.noname-b 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880b44c0 con 0x7f31880a98f0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: _finish_auth 0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: _check_auth_tickets 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3184001690 con 0x7f31880a98f0 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318c827640 10 monclient: dump: 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318fa8d640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:40:47.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.952+0000 7f318fa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31840017f0 con 0x7f31880a98f0 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318fa8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318fa8d640 4 set_mon_vals no callback set 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 1 -- 172.21.15.29:0/2359993723 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3184001c00 con 0x7f31880a98f0 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.964+0000 7f318c827640 10 monclient: dump: 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:47.964 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.965 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:47.965 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:47.965 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:47.965 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:47.965 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:47.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 10 monclient: shutdown 2024-03-25T19:40:47.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 1 -- 172.21.15.29:0/2359993723 >> v1:172.21.15.29:6790/0 conn(0x7f31880a98f0 legacy=0x7f31880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 1 -- 172.21.15.29:0/2359993723 shutdown_connections 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 1 -- 172.21.15.29:0/2359993723 >> 172.21.15.29:0/2359993723 conn(0x7f31880b7ea0 msgr2=0x7f31880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 1 -- 172.21.15.29:0/2359993723 shutdown_connections 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.965+0000 7f318fa8d640 1 -- 172.21.15.29:0/2359993723 wait complete. 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.966+0000 7f318fa8d640 10 monclient: build_initial_monmap 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.966+0000 7f318fa8d640 10 monclient: monmap: 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:47.967 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.966+0000 7f318fa8d640 1 librados: starting msgr at 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 1 librados: starting objecter 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 1 Processor -- start 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 1 -- start start 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 1 librados: setting wanted keys 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 1 librados: calling monclient init 2024-03-25T19:40:47.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.967+0000 7f318fa8d640 10 monclient: init 2024-03-25T19:40:47.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:47.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:47.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f31880a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:47.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f318813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:47.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f31881430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:47.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318813be70 con 0x7f31881430c0 2024-03-25T19:40:47.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188131660 con 0x7f318813f710 2024-03-25T19:40:47.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31881317f0 con 0x7f31880a98f0 2024-03-25T19:40:47.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.968+0000 7f318fa8d640 10 monclient(hunting): authenticate will time out at 6524.721298s 2024-03-25T19:40:47.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.980+0000 7f318d829640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f31880a98f0 0x7f3188130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:39110/0 (socket says 172.21.15.29:39110) 2024-03-25T19:40:47.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.980+0000 7f318d829640 1 -- 172.21.15.29:0/287789567 learned_addr learned my addr 172.21.15.29:0/287789567 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:47.985 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.985+0000 7f317e7fc640 10 client.?.objecter ms_handle_connect 0x7f31880a98f0 2024-03-25T19:40:47.985 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.985+0000 7f317e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.993 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.993+0000 7f317e7fc640 10 client.?.objecter ms_handle_connect 0x7f318813f710 2024-03-25T19:40:47.993 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.993+0000 7f317e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.997+0000 7f317e7fc640 10 client.?.objecter ms_handle_connect 0x7f31881430c0 2024-03-25T19:40:47.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.997+0000 7f317e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195906690 0 0) 0x7f31881317f0 con 0x7f31880a98f0 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:40:47.999 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:47.999+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f316c004eb0 con 0x7f31880a98f0 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 202779595 0 0) 0x7f316c004eb0 con 0x7f31880a98f0 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31881317f0 con 0x7f31880a98f0 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3184001bc0 con 0x7f31880a98f0 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:48.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:48.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.011+0000 7f317e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865964230 0 0) 0x7f3188131660 con 0x7f318813f710 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:40:48.013 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.013+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f316c004eb0 con 0x7f318813f710 2024-03-25T19:40:48.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261713640 0 0) 0x7f318813be70 con 0x7f31881430c0 2024-03-25T19:40:48.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.014 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:40:48.015 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.015+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3188131660 con 0x7f31881430c0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.027+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3176654976 0 0) 0x7f31881317f0 con 0x7f31880a98f0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.027+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 >> v1:172.21.15.196:6789/0 conn(0x7f318813f710 legacy=0x7f31881d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.027+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 >> v1:172.21.15.29:6789/0 conn(0x7f31881430c0 legacy=0x7f31881e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 1 monclient: found mon.c 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3188131980 con 0x7f31880a98f0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f317e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 10 monclient: _renew_subs 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3188135c60 con 0x7f31880a98f0 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 10 monclient: _renew_subs 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31880b7770 con 0x7f31880a98f0 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.028+0000 7f318fa8d640 1 librados: init done 2024-03-25T19:40:48.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.029+0000 7f318fa8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.029+0000 7f318fa8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.029+0000 7f318fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:48.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.029+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31881371d0 con 0x7f31880a98f0 2024-03-25T19:40:48.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31840011f0 con 0x7f31880a98f0 2024-03-25T19:40:48.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3184004c70 con 0x7f31880a98f0 2024-03-25T19:40:48.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317e7fc640 10 monclient: dump: 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:48.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.373+0000 7f317ffff640 4 set_mon_vals no callback set 2024-03-25T19:40:48.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.374+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f3184004e90 con 0x7f31880a98f0 2024-03-25T19:40:48.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.375+0000 7f318d028640 10 monclient: get_auth_request con 0x7f316c06de10 auth_method 0 2024-03-25T19:40:48.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.375+0000 7f317e7fc640 10 client.4535.objecter ms_handle_connect 0x7f316c06de10 2024-03-25T19:40:48.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f31840549e0 con 0x7f31880a98f0 2024-03-25T19:40:48.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 10 client.4535.objecter ms_dispatch 0x7f3188000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:48.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:48.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:48.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-25T19:40:48.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3184050320 con 0x7f31880a98f0 2024-03-25T19:40:48.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.380+0000 7f317e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:48.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.614+0000 7f318fa8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.614+0000 7f318fa8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.614+0000 7f318fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:48.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.614+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31880627b0 con 0x7f31880a98f0 2024-03-25T19:40:48.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.647+0000 7f317e7fc640 1 -- 172.21.15.29:0/287789567 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f3184052420 con 0x7f31880a98f0 2024-03-25T19:40:48.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.647+0000 7f317e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.647+0000 7f317e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:48.647 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:48.647 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:48.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 10 librados: watch_flush enter 2024-03-25T19:40:48.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 10 librados: watch_flush exit 2024-03-25T19:40:48.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-25T19:40:48.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 10 client.4535.objecter successfully canceled tick 2024-03-25T19:40:48.649 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 >> v1:172.21.15.29:6800/2314944989 conn(0x7f316c06de10 legacy=0x7f316c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 10 monclient: shutdown 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 >> v1:172.21.15.29:6790/0 conn(0x7f31880a98f0 legacy=0x7f3188130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 shutdown_connections 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 >> 172.21.15.29:0/287789567 conn(0x7f31880b7ea0 msgr2=0x7f3188074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 shutdown_connections 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 -- 172.21.15.29:0/287789567 wait complete. 2024-03-25T19:40:48.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:48.649+0000 7f318fa8d640 1 librados: shutdown 2024-03-25T19:40:48.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:50.796 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:20.959 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.802 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:20.966 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.850 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:21.013 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.858 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:21.021 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.866 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:21.029 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.877 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:21.039 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.881 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:21.043 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.889 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:21.052 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:50.907 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:21.069 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:55.466 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.581+0000 7fc471d4d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.581+0000 7fc471d4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.581+0000 7fc471d4d640 10 monclient: monmap: 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:40:55.582301+0000 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:40:55.582301+0000 2024-03-25T19:40:55.581 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.582 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:55.582 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:55.582 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:40:55.582 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:40:55.582 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.582+0000 7fc471d4d640 1 Processor -- start 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.582+0000 7fc471d4d640 1 -- start start 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.582+0000 7fc471d4d640 10 monclient: init 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fc46c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fc46c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:55.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fc46c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:55.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c142520 con 0x7fc46c1430c0 2024-03-25T19:40:55.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c0b28a0 con 0x7fc46c0a98f0 2024-03-25T19:40:55.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c0b2a30 con 0x7fc46c13f710 2024-03-25T19:40:55.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.584 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.583+0000 7fc471d4d640 10 monclient(hunting): authenticate will time out at 6532.336314s 2024-03-25T19:40:55.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.586+0000 7fc46b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc46c0a98f0 0x7fc46c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:48822/0 (socket says 172.21.15.29:48822) 2024-03-25T19:40:55.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.586+0000 7fc46b7fe640 1 -- 172.21.15.29:0/2639177595 learned_addr learned my addr 172.21.15.29:0/2639177595 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073942725 0 0) 0x7fc46c0b28a0 con 0x7fc46c0a98f0 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:40:55.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.614+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc454003520 con 0x7fc46c0a98f0 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603947813 0 0) 0x7fc46c0b2a30 con 0x7fc46c13f710 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:40:55.621 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.621+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c0b28a0 con 0x7fc46c13f710 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163560123 0 0) 0x7fc454003520 con 0x7fc46c0a98f0 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 >> v1:172.21.15.29:6790/0 conn(0x7fc46c13f710 legacy=0x7fc46c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 >> v1:172.21.15.29:6789/0 conn(0x7fc46c1430c0 legacy=0x7fc46c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c0b44c0 con 0x7fc46c0a98f0 2024-03-25T19:40:55.625 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc45c001690 con 0x7fc46c0a98f0 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc46a7fc640 10 monclient: dump: 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.626 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc471d4d640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:40:55.627 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.625+0000 7fc471d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.642+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc45c0017f0 con 0x7fc46c0a98f0 2024-03-25T19:40:55.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.642+0000 7fc46a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.642+0000 7fc46a7fc640 1 -- 172.21.15.29:0/2639177595 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc45c001c00 con 0x7fc46c0a98f0 2024-03-25T19:40:55.642 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.642+0000 7fc471d4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.642+0000 7fc471d4d640 4 set_mon_vals no callback set 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc46a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc46a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc46a7fc640 10 monclient: dump: 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:55.643 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 10 monclient: shutdown 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 1 -- 172.21.15.29:0/2639177595 >> v1:172.21.15.196:6789/0 conn(0x7fc46c0a98f0 legacy=0x7fc46c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 1 -- 172.21.15.29:0/2639177595 shutdown_connections 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 1 -- 172.21.15.29:0/2639177595 >> 172.21.15.29:0/2639177595 conn(0x7fc46c0b7ea0 msgr2=0x7fc46c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 1 -- 172.21.15.29:0/2639177595 shutdown_connections 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.643+0000 7fc471d4d640 1 -- 172.21.15.29:0/2639177595 wait complete. 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.644+0000 7fc471d4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.644+0000 7fc471d4d640 10 monclient: monmap: 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:55.646 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.644+0000 7fc471d4d640 1 librados: starting msgr at 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.644+0000 7fc471d4d640 1 librados: starting objecter 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.645+0000 7fc471d4d640 1 Processor -- start 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.645+0000 7fc471d4d640 1 -- start start 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.645+0000 7fc471d4d640 1 librados: setting wanted keys 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.645+0000 7fc471d4d640 1 librados: calling monclient init 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.645+0000 7fc471d4d640 10 monclient: init 2024-03-25T19:40:55.647 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fc46c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fc46c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fc46c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c139a40 con 0x7fc46c1430c0 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c133f20 con 0x7fc46c0a98f0 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c0ab950 con 0x7fc46c13f710 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.648 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.646+0000 7fc471d4d640 10 monclient(hunting): authenticate will time out at 6532.399139s 2024-03-25T19:40:55.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.665+0000 7fc46affd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fc46c13f710 0x7fc46c133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:43644/0 (socket says 172.21.15.29:43644) 2024-03-25T19:40:55.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.665+0000 7fc46affd640 1 -- 172.21.15.29:0/1567536533 learned_addr learned my addr 172.21.15.29:0/1567536533 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:40:55.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.667+0000 7fc453fff640 10 client.?.objecter ms_handle_connect 0x7fc46c13f710 2024-03-25T19:40:55.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.667+0000 7fc453fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.674+0000 7fc453fff640 10 client.?.objecter ms_handle_connect 0x7fc46c1430c0 2024-03-25T19:40:55.673 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.674+0000 7fc453fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.675+0000 7fc453fff640 10 client.?.objecter ms_handle_connect 0x7fc46c0a98f0 2024-03-25T19:40:55.674 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.675+0000 7fc453fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290353254 0 0) 0x7fc46c0ab950 con 0x7fc46c13f710 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:40:55.680 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.680+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc458004eb0 con 0x7fc46c13f710 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942791694 0 0) 0x7fc46c139a40 con 0x7fc46c1430c0 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.684+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c0ab950 con 0x7fc46c1430c0 2024-03-25T19:40:55.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 486017875 0 0) 0x7fc458004eb0 con 0x7fc46c13f710 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc46c139a40 con 0x7fc46c13f710 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc460001a10 con 0x7fc46c13f710 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 10 monclient(hunting): dump: 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:55.694 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:55.695 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.695 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.694+0000 7fc453fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725794564 0 0) 0x7fc46c133f20 con 0x7fc46c0a98f0 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.697+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc458004eb0 con 0x7fc46c0a98f0 2024-03-25T19:40:55.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1071956816 0 0) 0x7fc46c0ab950 con 0x7fc46c1430c0 2024-03-25T19:40:55.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc46c133f20 con 0x7fc46c1430c0 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc464001b00 con 0x7fc46c1430c0 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.714+0000 7fc453fff640 10 monclient(hunting): dump: 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.722+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3853117490 0 0) 0x7fc46c139a40 con 0x7fc46c13f710 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.722+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 >> v1:172.21.15.196:6789/0 conn(0x7fc46c0a98f0 legacy=0x7fc46c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.722+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 >> v1:172.21.15.29:6789/0 conn(0x7fc46c1430c0 legacy=0x7fc46c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 1 monclient: found mon.c 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c0abc30 con 0x7fc46c13f710 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 10 monclient: _renew_subs 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c0abf10 con 0x7fc46c13f710 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 10 monclient: _renew_subs 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c139370 con 0x7fc46c13f710 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.723+0000 7fc471d4d640 1 librados: init done 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.724+0000 7fc471d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.724+0000 7fc471d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.724+0000 7fc471d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:55.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:55.724+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46c0627b0 con 0x7fc46c13f710 2024-03-25T19:40:56.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc460001c50 con 0x7fc46c13f710 2024-03-25T19:40:56.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:56.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc460003a60 con 0x7fc46c13f710 2024-03-25T19:40:56.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc453fff640 10 monclient: dump: 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:40:56.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.037+0000 7fc469ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:56.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.038+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fc460003c80 con 0x7fc46c13f710 2024-03-25T19:40:56.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.039+0000 7fc46b7fe640 10 monclient: get_auth_request con 0x7fc45806dfb0 auth_method 0 2024-03-25T19:40:56.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.039+0000 7fc453fff640 10 client.4541.objecter ms_handle_connect 0x7fc45806dfb0 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fc460054830 con 0x7fc46c13f710 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 10 client.4541.objecter ms_dispatch 0x7fc46c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.044+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc460050340 con 0x7fc46c13f710 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.045+0000 7fc453fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:56.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.045+0000 7fc453fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:56.295 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.296+0000 7fc471d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.296+0000 7fc471d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.296+0000 7fc471d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:40:56.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.296+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc46c0b7180 con 0x7fc46c13f710 2024-03-25T19:40:56.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.326+0000 7fc453fff640 1 -- 172.21.15.29:0/1567536533 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fc460052270 con 0x7fc46c13f710 2024-03-25T19:40:56.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.326+0000 7fc453fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.326+0000 7fc453fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:56.326 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:40:56.326 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:56.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 10 librados: watch_flush enter 2024-03-25T19:40:56.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 10 librados: watch_flush exit 2024-03-25T19:40:56.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T19:40:56.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 10 client.4541.objecter successfully canceled tick 2024-03-25T19:40:56.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 >> v1:172.21.15.29:6800/2314944989 conn(0x7fc45806dfb0 legacy=0x7fc458070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 10 monclient: shutdown 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 >> v1:172.21.15.29:6790/0 conn(0x7fc46c13f710 legacy=0x7fc46c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.328+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 shutdown_connections 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.329+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 >> 172.21.15.29:0/1567536533 conn(0x7fc46c0b7ea0 msgr2=0x7fc46c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.329+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 shutdown_connections 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.329+0000 7fc471d4d640 1 -- 172.21.15.29:0/1567536533 wait complete. 2024-03-25T19:40:56.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:40:56.329+0000 7fc471d4d640 1 librados: shutdown 2024-03-25T19:40:56.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:03.147 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.261+0000 7f54e4372640 10 monclient: get_monmap_and_config 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.261+0000 7f54e4372640 10 monclient: build_initial_monmap 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.261+0000 7f54e4372640 10 monclient: monmap: 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:03.262346+0000 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:03.262346+0000 2024-03-25T19:41:03.261 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:03.262 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:03.262 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:03.262 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:03.262 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:03.262 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.262+0000 7f54e4372640 1 Processor -- start 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.262+0000 7f54e4372640 1 -- start start 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.262+0000 7f54e4372640 10 monclient: init 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-a con 0x7f54dc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-b con 0x7f54dc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc142520 con 0x7f54dc0a98f0 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0b28a0 con 0x7f54dc1430c0 2024-03-25T19:41:03.263 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0b2a30 con 0x7f54dc13f710 2024-03-25T19:41:03.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:03.264 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.263+0000 7f54e4372640 10 monclient(hunting): authenticate will time out at 6540.016079s 2024-03-25T19:41:03.265 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.266+0000 7f54e290f640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54dc1430c0 0x7f54dc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:37642/0 (socket says 172.21.15.29:37642) 2024-03-25T19:41:03.265 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.266+0000 7f54e290f640 1 -- 172.21.15.29:0/1377849194 learned_addr learned my addr 172.21.15.29:0/1377849194 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278162037 0 0) 0x7f54dc142520 con 0x7f54dc0a98f0 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:41:03.282 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.282+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c8003520 con 0x7f54dc0a98f0 2024-03-25T19:41:03.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764062076 0 0) 0x7f54dc0b28a0 con 0x7f54dc1430c0 2024-03-25T19:41:03.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:41:03.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.288+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc142520 con 0x7f54dc1430c0 2024-03-25T19:41:03.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994017068 0 0) 0x7f54dc0b2a30 con 0x7f54dc13f710 2024-03-25T19:41:03.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.290 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:41:03.291 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.291+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc0b28a0 con 0x7f54dc13f710 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.308+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910959770 0 0) 0x7f54c8003520 con 0x7f54dc0a98f0 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.308+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 >> v1:172.21.15.29:6790/0 conn(0x7f54dc13f710 legacy=0x7f54dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 >> v1:172.21.15.196:6789/0 conn(0x7f54dc1430c0 legacy=0x7f54dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: _finish_hunting 0 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 1 monclient: found mon.noname-a 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0b44c0 con 0x7f54dc0a98f0 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: _finish_auth 0 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: _check_auth_tickets 2024-03-25T19:41:03.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f54d8001690 con 0x7f54dc0a98f0 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e110c640 10 monclient: dump: 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:03.310 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e4372640 5 monclient: authenticate success, global_id 4545 2024-03-25T19:41:03.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.309+0000 7f54e4372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54d80017f0 con 0x7f54dc0a98f0 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 1 -- 172.21.15.29:0/1377849194 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f54d8001c00 con 0x7f54dc0a98f0 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e110c640 10 monclient: dump: 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.313 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e4372640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:03.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.313+0000 7f54e4372640 4 set_mon_vals no callback set 2024-03-25T19:41:03.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 10 monclient: shutdown 2024-03-25T19:41:03.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 1 -- 172.21.15.29:0/1377849194 >> v1:172.21.15.29:6789/0 conn(0x7f54dc0a98f0 legacy=0x7f54dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 1 -- 172.21.15.29:0/1377849194 shutdown_connections 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 1 -- 172.21.15.29:0/1377849194 >> 172.21.15.29:0/1377849194 conn(0x7f54dc0b7ea0 msgr2=0x7f54dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 1 -- 172.21.15.29:0/1377849194 shutdown_connections 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.314+0000 7f54e4372640 1 -- 172.21.15.29:0/1377849194 wait complete. 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.315+0000 7f54e4372640 10 monclient: build_initial_monmap 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.315+0000 7f54e4372640 10 monclient: monmap: 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.315+0000 7f54e4372640 1 librados: starting msgr at 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.315+0000 7f54e4372640 1 librados: starting objecter 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 Processor -- start 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 -- start start 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 librados: setting wanted keys 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 librados: calling monclient init 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient: init 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-a con 0x7f54dc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-b con 0x7f54dc13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc137320 con 0x7f54dc0a98f0 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc137090 con 0x7f54dc13f710 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc1375c0 con 0x7f54dc1430c0 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.316+0000 7f54e4372640 10 monclient(hunting): authenticate will time out at 6540.069739s 2024-03-25T19:41:03.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.328+0000 7f54e190d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54dc13f710 0x7f54dc1d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:37676/0 (socket says 172.21.15.29:37676) 2024-03-25T19:41:03.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.328+0000 7f54e190d640 1 -- 172.21.15.29:0/990764311 learned_addr learned my addr 172.21.15.29:0/990764311 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:03.332 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.332+0000 7f54d2ffd640 10 client.?.objecter ms_handle_connect 0x7f54dc13f710 2024-03-25T19:41:03.332 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.332+0000 7f54d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.338+0000 7f54d2ffd640 10 client.?.objecter ms_handle_connect 0x7f54dc1430c0 2024-03-25T19:41:03.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.338+0000 7f54d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.339+0000 7f54d2ffd640 10 client.?.objecter ms_handle_connect 0x7f54dc0a98f0 2024-03-25T19:41:03.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.339+0000 7f54d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419457191 0 0) 0x7f54dc137090 con 0x7f54dc13f710 2024-03-25T19:41:03.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.342+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c0004eb0 con 0x7f54dc13f710 2024-03-25T19:41:03.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831924610 0 0) 0x7f54dc137320 con 0x7f54dc0a98f0 2024-03-25T19:41:03.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:41:03.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.349+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc137090 con 0x7f54dc0a98f0 2024-03-25T19:41:03.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461945835 0 0) 0x7f54dc1375c0 con 0x7f54dc1430c0 2024-03-25T19:41:03.349 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:41:03.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.350+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc137320 con 0x7f54dc1430c0 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578741891 0 0) 0x7f54c0004eb0 con 0x7f54dc13f710 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54dc1375c0 con 0x7f54dc13f710 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f54cc001a10 con 0x7f54dc13f710 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.361 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:03.362 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:03.362 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:03.362 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:03.362 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.362 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.361+0000 7f54d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 769488131 0 0) 0x7f54dc1375c0 con 0x7f54dc13f710 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 >> v1:172.21.15.29:6790/0 conn(0x7f54dc1430c0 legacy=0x7f54dc133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 >> v1:172.21.15.29:6789/0 conn(0x7f54dc0a98f0 legacy=0x7f54dc136980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 1 monclient: found mon.b 2024-03-25T19:41:03.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc1d7b00 con 0x7f54dc13f710 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 10 monclient: _renew_subs 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc137ae0 con 0x7f54dc13f710 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 10 monclient: _renew_subs 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc0b7330 con 0x7f54dc13f710 2024-03-25T19:41:03.372 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.371+0000 7f54e4372640 1 librados: init done 2024-03-25T19:41:03.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.372+0000 7f54e4372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.372+0000 7f54e4372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.372+0000 7f54e4372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:03.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.372+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc137750 con 0x7f54dc13f710 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54cc001c70 con 0x7f54dc13f710 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f54cc003a60 con 0x7f54dc13f710 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54e090b640 4 set_mon_vals no callback set 2024-03-25T19:41:03.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.657+0000 7f54d2ffd640 10 monclient: dump: 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:03.658 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:03.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.658+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f54cc003c80 con 0x7f54dc13f710 2024-03-25T19:41:03.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.659+0000 7f54e210e640 10 monclient: get_auth_request con 0x7f54c006ddf0 auth_method 0 2024-03-25T19:41:03.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.659+0000 7f54d2ffd640 10 client.4549.objecter ms_handle_connect 0x7f54c006ddf0 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f54cc054860 con 0x7f54dc13f710 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 10 client.4549.objecter ms_dispatch 0x7f54dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.663+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54cc050370 con 0x7f54dc13f710 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.664+0000 7f54d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.663 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.664+0000 7f54d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:03.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.929+0000 7f54e4372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.929+0000 7f54e4372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.929+0000 7f54e4372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:03.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:03.929+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54dc139150 con 0x7f54dc13f710 2024-03-25T19:41:04.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.064+0000 7f54d2ffd640 1 -- 172.21.15.29:0/990764311 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f54cc0522a0 con 0x7f54dc13f710 2024-03-25T19:41:04.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.064+0000 7f54d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:04.064 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.064+0000 7f54d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:04.064 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:04.064 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:04.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 10 librados: watch_flush enter 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 10 librados: watch_flush exit 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 10 client.4549.objecter successfully canceled tick 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 >> v1:172.21.15.29:6800/2314944989 conn(0x7f54c006ddf0 legacy=0x7f54c00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 10 monclient: shutdown 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 >> v1:172.21.15.196:6789/0 conn(0x7f54dc13f710 legacy=0x7f54dc1d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.066+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 shutdown_connections 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.067+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 >> 172.21.15.29:0/990764311 conn(0x7f54dc0b7ea0 msgr2=0x7f54dc13ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.067+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 shutdown_connections 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.067+0000 7f54e4372640 1 -- 172.21.15.29:0/990764311 wait complete. 2024-03-25T19:41:04.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:04.067+0000 7f54e4372640 1 librados: shutdown 2024-03-25T19:41:04.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:10.856 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:41.019 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.863 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:41.026 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.883 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:10.898 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:41.061 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.916 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:41.078 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.920 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:41.083 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.924 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:41.087 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.939 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:41.103 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.957 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:07:41.119 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.967 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:41.131 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:10.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.991+0000 7f2652cd9640 10 monclient: get_monmap_and_config 2024-03-25T19:41:10.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.991+0000 7f2652cd9640 10 monclient: build_initial_monmap 2024-03-25T19:41:10.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.991+0000 7f2652cd9640 10 monclient: monmap: 2024-03-25T19:41:10.992 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:10.992 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:10.993397+0000 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:10.993397+0000 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:10.993 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.992+0000 7f2652cd9640 1 Processor -- start 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.992+0000 7f2652cd9640 1 -- start start 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.992+0000 7f2652cd9640 10 monclient: init 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f264c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f264c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f264c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c142520 con 0x7f264c1430c0 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0b28a0 con 0x7f264c13f710 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0b2a30 con 0x7f264c0a98f0 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:10.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:10.993+0000 7f2652cd9640 10 monclient(hunting): authenticate will time out at 6547.747138s 2024-03-25T19:41:11.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.010+0000 7f2650a75640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f264c0a98f0 0x7f264c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:42782/0 (socket says 172.21.15.29:42782) 2024-03-25T19:41:11.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.010+0000 7f2650a75640 1 -- 172.21.15.29:0/446935260 learned_addr learned my addr 172.21.15.29:0/446935260 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173793832 0 0) 0x7f264c0b28a0 con 0x7f264c13f710 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:41:11.039 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.038+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c003520 con 0x7f264c13f710 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952335958 0 0) 0x7f264c0b2a30 con 0x7f264c0a98f0 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:41:11.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.056+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c0b28a0 con 0x7f264c0a98f0 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263153270 0 0) 0x7f264c142520 con 0x7f264c1430c0 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:41:11.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.064+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c0b2a30 con 0x7f264c1430c0 2024-03-25T19:41:11.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762028050 0 0) 0x7f263c003520 con 0x7f264c13f710 2024-03-25T19:41:11.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 >> v1:172.21.15.29:6790/0 conn(0x7f264c0a98f0 legacy=0x7f264c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 >> v1:172.21.15.29:6789/0 conn(0x7f264c1430c0 legacy=0x7f264c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0b44c0 con 0x7f264c13f710 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2638001690 con 0x7f264c13f710 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f264b7fe640 10 monclient: dump: 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.068 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f2652cd9640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:41:11.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.066+0000 7f2652cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26380017f0 con 0x7f264c13f710 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 1 -- 172.21.15.29:0/446935260 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2638001c00 con 0x7f264c13f710 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f264b7fe640 10 monclient: dump: 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:11.075 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f2652cd9640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:11.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.074+0000 7f2652cd9640 4 set_mon_vals no callback set 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 10 monclient: shutdown 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 1 -- 172.21.15.29:0/446935260 >> v1:172.21.15.196:6789/0 conn(0x7f264c13f710 legacy=0x7f264c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 1 -- 172.21.15.29:0/446935260 shutdown_connections 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 1 -- 172.21.15.29:0/446935260 >> 172.21.15.29:0/446935260 conn(0x7f264c0b7ea0 msgr2=0x7f264c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:11.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 1 -- 172.21.15.29:0/446935260 shutdown_connections 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.075+0000 7f2652cd9640 1 -- 172.21.15.29:0/446935260 wait complete. 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 10 monclient: build_initial_monmap 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 10 monclient: monmap: 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 librados: starting msgr at 2024-03-25T19:41:11.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 librados: starting objecter 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 Processor -- start 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 -- start start 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 librados: setting wanted keys 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 1 librados: calling monclient init 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.076+0000 7f2652cd9640 10 monclient: init 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f264c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f264c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f264c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c138250 con 0x7f264c1430c0 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c137d50 con 0x7f264c0a98f0 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c137ee0 con 0x7f264c13f710 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:11.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.077+0000 7f2652cd9640 10 monclient(hunting): authenticate will time out at 6547.831498s 2024-03-25T19:41:11.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.090+0000 7f264bfff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f264c13f710 0x7f264c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:42834/0 (socket says 172.21.15.29:42834) 2024-03-25T19:41:11.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.090+0000 7f264bfff640 1 -- 172.21.15.29:0/3337504067 learned_addr learned my addr 172.21.15.29:0/3337504067 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.092+0000 7f26497fa640 10 client.?.objecter ms_handle_connect 0x7f264c13f710 2024-03-25T19:41:11.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.092+0000 7f26497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:11.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.113+0000 7f26497fa640 10 client.?.objecter ms_handle_connect 0x7f264c1430c0 2024-03-25T19:41:11.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.113+0000 7f26497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205708132 0 0) 0x7f264c137ee0 con 0x7f264c13f710 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:41:11.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.122+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2630004eb0 con 0x7f264c13f710 2024-03-25T19:41:11.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.124+0000 7f26497fa640 10 client.?.objecter ms_handle_connect 0x7f264c0a98f0 2024-03-25T19:41:11.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.124+0000 7f26497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:11.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529409316 0 0) 0x7f264c138250 con 0x7f264c1430c0 2024-03-25T19:41:11.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:41:11.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.130+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c137ee0 con 0x7f264c1430c0 2024-03-25T19:41:11.153 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053081975 0 0) 0x7f264c137d50 con 0x7f264c0a98f0 2024-03-25T19:41:11.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:11.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:11.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:11.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:41:11.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.153+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c138250 con 0x7f264c0a98f0 2024-03-25T19:41:11.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2591608156 0 0) 0x7f264c137ee0 con 0x7f264c1430c0 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f264c137d50 con 0x7f264c1430c0 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2644001b50 con 0x7f264c1430c0 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 10 monclient(hunting): dump: 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:11.157 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:11.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.156+0000 7f26497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3813943881 0 0) 0x7f264c137d50 con 0x7f264c1430c0 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 >> v1:172.21.15.29:6790/0 conn(0x7f264c13f710 legacy=0x7f264c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 >> v1:172.21.15.196:6789/0 conn(0x7f264c0a98f0 legacy=0x7f264c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 1 monclient: found mon.a 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:11.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0b7730 con 0x7f264c1430c0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f26497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 10 monclient: _renew_subs 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264c0b7920 con 0x7f264c1430c0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 10 monclient: _renew_subs 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c1e33d0 con 0x7f264c1430c0 2024-03-25T19:41:11.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.165+0000 7f2652cd9640 1 librados: init done 2024-03-25T19:41:11.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.166+0000 7f2652cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.166+0000 7f2652cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.166+0000 7f2652cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:11.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.166+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c0627b0 con 0x7f264c1430c0 2024-03-25T19:41:11.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2644001fe0 con 0x7f264c1430c0 2024-03-25T19:41:11.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2644003b30 con 0x7f264c1430c0 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f26497fa640 10 monclient: dump: 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:11.482 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.480+0000 7f264affd640 4 set_mon_vals no callback set 2024-03-25T19:41:11.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.481+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f2644003d50 con 0x7f264c1430c0 2024-03-25T19:41:11.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.482+0000 7f2650a75640 10 monclient: get_auth_request con 0x7f263006ddd0 auth_method 0 2024-03-25T19:41:11.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.482+0000 7f26497fa640 10 client.4554.objecter ms_handle_connect 0x7f263006ddd0 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f2644053840 con 0x7f264c1430c0 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 10 client.4554.objecter ms_dispatch 0x7f264c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f264404f350 con 0x7f264c1430c0 2024-03-25T19:41:11.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.489+0000 7f26497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:11.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.741+0000 7f2652cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.741+0000 7f2652cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.741+0000 7f2652cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:11.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.741+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f264c138400 con 0x7f264c1430c0 2024-03-25T19:41:11.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.777+0000 7f26497fa640 1 -- 172.21.15.29:0/3337504067 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f264c138400 con 0x7f264c1430c0 2024-03-25T19:41:11.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.777+0000 7f26497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.777+0000 7f26497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:11.778 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:11.778 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 10 librados: watch_flush enter 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 10 librados: watch_flush exit 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 10 client.4554.objecter successfully canceled tick 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 >> v1:172.21.15.29:6800/2314944989 conn(0x7f263006ddd0 legacy=0x7f2630070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.780 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 10 monclient: shutdown 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 >> v1:172.21.15.29:6789/0 conn(0x7f264c1430c0 legacy=0x7f264c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 shutdown_connections 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 >> 172.21.15.29:0/3337504067 conn(0x7f264c0b7ea0 msgr2=0x7f264c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 shutdown_connections 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 -- 172.21.15.29:0/3337504067 wait complete. 2024-03-25T19:41:11.781 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:11.779+0000 7f2652cd9640 1 librados: shutdown 2024-03-25T19:41:11.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:18.599 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:18.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.715+0000 7f415a9c8640 10 monclient: get_monmap_and_config 2024-03-25T19:41:18.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.715+0000 7f415a9c8640 10 monclient: build_initial_monmap 2024-03-25T19:41:18.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.715+0000 7f415a9c8640 10 monclient: monmap: 2024-03-25T19:41:18.715 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:18.716524+0000 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:18.716524+0000 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:18.716 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.716+0000 7f415a9c8640 1 Processor -- start 2024-03-25T19:41:18.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.716+0000 7f415a9c8640 1 -- start start 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.716+0000 7f415a9c8640 10 monclient: init 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f41540a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f415413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f41541430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4154142520 con 0x7f41540a98f0 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540b28a0 con 0x7f41541430c0 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540b2a30 con 0x7f415413f710 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:18.718 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.717+0000 7f415a9c8640 10 monclient(hunting): authenticate will time out at 6555.470652s 2024-03-25T19:41:18.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.721+0000 7f4158f65640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f41541430c0 0x7f415414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:40556/0 (socket says 172.21.15.29:40556) 2024-03-25T19:41:18.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.721+0000 7f4158f65640 1 -- 172.21.15.29:0/4012220104 learned_addr learned my addr 172.21.15.29:0/4012220104 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:18.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746460105 0 0) 0x7f41540b28a0 con 0x7f41541430c0 2024-03-25T19:41:18.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:41:18.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.741+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c003520 con 0x7f41541430c0 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409938988 0 0) 0x7f4154142520 con 0x7f41540a98f0 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:41:18.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.750+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41540b28a0 con 0x7f41540a98f0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 872705156 0 0) 0x7f413c003520 con 0x7f41541430c0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 >> v1:172.21.15.29:6790/0 conn(0x7f415413f710 legacy=0x7f41541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 >> v1:172.21.15.29:6789/0 conn(0x7f41540a98f0 legacy=0x7f41540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41540b44c0 con 0x7f41541430c0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:18.765 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f414c001a10 con 0x7f41541430c0 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f4152ffd640 10 monclient: dump: 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f415a9c8640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:41:18.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.765+0000 7f415a9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c001b70 con 0x7f41541430c0 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 1 -- 172.21.15.29:0/4012220104 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f414c001f80 con 0x7f41541430c0 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f415a9c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f415a9c8640 4 set_mon_vals no callback set 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.769+0000 7f4152ffd640 10 monclient: dump: 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.769 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:18.770 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 10 monclient: shutdown 2024-03-25T19:41:18.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:18.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 1 -- 172.21.15.29:0/4012220104 >> v1:172.21.15.196:6789/0 conn(0x7f41541430c0 legacy=0x7f415414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 1 -- 172.21.15.29:0/4012220104 shutdown_connections 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 1 -- 172.21.15.29:0/4012220104 >> 172.21.15.29:0/4012220104 conn(0x7f41540b7ea0 msgr2=0x7f41540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 1 -- 172.21.15.29:0/4012220104 shutdown_connections 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.770+0000 7f415a9c8640 1 -- 172.21.15.29:0/4012220104 wait complete. 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.771+0000 7f415a9c8640 10 monclient: build_initial_monmap 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.771+0000 7f415a9c8640 10 monclient: monmap: 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:18.773 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.771+0000 7f415a9c8640 1 librados: starting msgr at 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.771+0000 7f415a9c8640 1 librados: starting objecter 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.771+0000 7f415a9c8640 1 Processor -- start 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 -- start start 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 librados: setting wanted keys 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 librados: calling monclient init 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient: init 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f41540a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f415413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f41541430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415413c2b0 con 0x7f41541430c0 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4154136f10 con 0x7f41540a98f0 2024-03-25T19:41:18.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41541370a0 con 0x7f415413f710 2024-03-25T19:41:18.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:18.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.772+0000 7f415a9c8640 10 monclient(hunting): authenticate will time out at 6555.525483s 2024-03-25T19:41:18.782 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.782+0000 7f41537fe640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f415413f710 0x7f41541377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:40420/0 (socket says 172.21.15.29:40420) 2024-03-25T19:41:18.782 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.783+0000 7f41537fe640 1 -- 172.21.15.29:0/794512922 learned_addr learned my addr 172.21.15.29:0/794512922 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:18.783 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.783+0000 7f4150ff9640 10 client.?.objecter ms_handle_connect 0x7f415413f710 2024-03-25T19:41:18.783 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.783+0000 7f4150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.786 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.787+0000 7f4150ff9640 10 client.?.objecter ms_handle_connect 0x7f41540a98f0 2024-03-25T19:41:18.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.787+0000 7f4150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954336360 0 0) 0x7f41541370a0 con 0x7f415413f710 2024-03-25T19:41:18.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.790 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.790 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:41:18.790 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.790+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140004eb0 con 0x7f415413f710 2024-03-25T19:41:18.790 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.791+0000 7f4150ff9640 10 client.?.objecter ms_handle_connect 0x7f41541430c0 2024-03-25T19:41:18.790 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.791+0000 7f4150ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288521442 0 0) 0x7f4154136f10 con 0x7f41540a98f0 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:41:18.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.796+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41541370a0 con 0x7f41540a98f0 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256205955 0 0) 0x7f415413c2b0 con 0x7f41541430c0 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:41:18.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.797+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154136f10 con 0x7f41541430c0 2024-03-25T19:41:18.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536035233 0 0) 0x7f4154136f10 con 0x7f41541430c0 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415413c2b0 con 0x7f41541430c0 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f414c001e80 con 0x7f41541430c0 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.809 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:18.810 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:18.810 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:18.810 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.809+0000 7f4150ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389945219 0 0) 0x7f4140004eb0 con 0x7f415413f710 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4154136f10 con 0x7f415413f710 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f41480019d0 con 0x7f415413f710 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.810+0000 7f4150ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:18.811 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270217500 0 0) 0x7f41541370a0 con 0x7f41540a98f0 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4140004eb0 con 0x7f41540a98f0 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4144001930 con 0x7f41540a98f0 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.814+0000 7f4150ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:18.814 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:18.815 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:18.815 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:18.815 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1507344931 0 0) 0x7f415413c2b0 con 0x7f41541430c0 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 >> v1:172.21.15.29:6790/0 conn(0x7f415413f710 legacy=0x7f41541377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 >> v1:172.21.15.196:6789/0 conn(0x7f41540a98f0 legacy=0x7f41541d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 1 monclient: found mon.a 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154137230 con 0x7f41541430c0 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:18.819 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f4150ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 10 monclient: _renew_subs 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415413b620 con 0x7f41541430c0 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 10 monclient: _renew_subs 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4154133710 con 0x7f41541430c0 2024-03-25T19:41:18.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.819+0000 7f415a9c8640 1 librados: init done 2024-03-25T19:41:18.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.820+0000 7f415a9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:18.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.820+0000 7f415a9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:18.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.820+0000 7f415a9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:18.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:18.820+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41540627b0 con 0x7f41541430c0 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c002000 con 0x7f41541430c0 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f414c0015d0 con 0x7f41541430c0 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f4150ff9640 10 monclient: dump: 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:19.135 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.135+0000 7f41527fc640 4 set_mon_vals no callback set 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.136+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f414c01acc0 con 0x7f41541430c0 2024-03-25T19:41:19.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.136+0000 7f4153fff640 10 monclient: get_auth_request con 0x7f414006e0f0 auth_method 0 2024-03-25T19:41:19.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.136+0000 7f4150ff9640 10 client.4560.objecter ms_handle_connect 0x7f414006e0f0 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f414c0548e0 con 0x7f41541430c0 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 10 client.4560.objecter ms_dispatch 0x7f4154000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f414c050220 con 0x7f41541430c0 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:19.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.142+0000 7f4150ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:19.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.397+0000 7f415a9c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.398+0000 7f415a9c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.398+0000 7f415a9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:19.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.398+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41540008d0 con 0x7f41541430c0 2024-03-25T19:41:19.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.433+0000 7f4150ff9640 1 -- 172.21.15.29:0/794512922 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f414c052320 con 0x7f41541430c0 2024-03-25T19:41:19.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.433+0000 7f4150ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.433+0000 7f4150ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:19.433 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:19.433 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:19.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 10 librados: watch_flush enter 2024-03-25T19:41:19.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 10 librados: watch_flush exit 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 10 client.4560.objecter successfully canceled tick 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 >> v1:172.21.15.29:6800/2314944989 conn(0x7f414006e0f0 legacy=0x7f41400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 10 monclient: shutdown 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 >> v1:172.21.15.29:6789/0 conn(0x7f41541430c0 legacy=0x7f4154133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 shutdown_connections 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.435+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 >> 172.21.15.29:0/794512922 conn(0x7f41540b7ea0 msgr2=0x7f4154074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.436+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 shutdown_connections 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.436+0000 7f415a9c8640 1 -- 172.21.15.29:0/794512922 wait complete. 2024-03-25T19:41:19.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:19.436+0000 7f415a9c8640 1 librados: shutdown 2024-03-25T19:41:19.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:26.255 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:26.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.378+0000 7f993528f640 10 monclient: get_monmap_and_config 2024-03-25T19:41:26.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.378+0000 7f993528f640 10 monclient: build_initial_monmap 2024-03-25T19:41:26.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.378+0000 7f993528f640 10 monclient: monmap: 2024-03-25T19:41:26.378 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:26.379651+0000 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:26.379651+0000 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:26.379 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.379+0000 7f993528f640 1 Processor -- start 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.379+0000 7f993528f640 1 -- start start 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.379+0000 7f993528f640 10 monclient: init 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient(hunting): picked mon.noname-a con 0x7f99300705b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient(hunting): picked mon.noname-b con 0x7f99301402d0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:26.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient(hunting): picked mon.noname-c con 0x7f9930149ae0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:26.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300718c0 con 0x7f99300705b0 2024-03-25T19:41:26.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300714c0 con 0x7f9930149ae0 2024-03-25T19:41:26.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9930072b50 con 0x7f99301402d0 2024-03-25T19:41:26.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:26.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.380+0000 7f993528f640 10 monclient(hunting): authenticate will time out at 6563.133687s 2024-03-25T19:41:26.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.391+0000 7f992f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9930149ae0 0x7f993014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:37084/0 (socket says 172.21.15.29:37084) 2024-03-25T19:41:26.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.391+0000 7f992f7fe640 1 -- 172.21.15.29:0/1153683665 learned_addr learned my addr 172.21.15.29:0/1153683665 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290858910 0 0) 0x7f99300714c0 con 0x7f9930149ae0 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918003520 con 0x7f9930149ae0 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552269199 0 0) 0x7f9930072b50 con 0x7f99301402d0 2024-03-25T19:41:26.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:41:26.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.407+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300714c0 con 0x7f99301402d0 2024-03-25T19:41:26.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957480082 0 0) 0x7f99300718c0 con 0x7f99300705b0 2024-03-25T19:41:26.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:41:26.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.417+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930072b50 con 0x7f99300705b0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203654205 0 0) 0x7f9918003520 con 0x7f9930149ae0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 >> v1:172.21.15.29:6790/0 conn(0x7f99301402d0 legacy=0x7f9930140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 >> v1:172.21.15.29:6789/0 conn(0x7f99300705b0 legacy=0x7f993013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 1 monclient: found mon.noname-c 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993013dd10 con 0x7f9930149ae0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:26.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f992dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:26.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f993528f640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:41:26.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.420+0000 7f993528f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f99280019d0 con 0x7f9930149ae0 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: dump: 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:26.428 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9928001c20 con 0x7f9930149ae0 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f993528f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 1 -- 172.21.15.29:0/1153683665 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9928001ff0 con 0x7f9930149ae0 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.428+0000 7f992dffb640 10 monclient: dump: 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.429+0000 7f993528f640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:26.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.429+0000 7f993528f640 4 set_mon_vals no callback set 2024-03-25T19:41:26.432 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.429+0000 7f993528f640 10 monclient: shutdown 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.429+0000 7f993528f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.429+0000 7f993528f640 1 -- 172.21.15.29:0/1153683665 >> v1:172.21.15.196:6789/0 conn(0x7f9930149ae0 legacy=0x7f993014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.430+0000 7f993528f640 1 -- 172.21.15.29:0/1153683665 shutdown_connections 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.430+0000 7f993528f640 1 -- 172.21.15.29:0/1153683665 >> 172.21.15.29:0/1153683665 conn(0x7f99300b5d10 msgr2=0x7f99300b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.430+0000 7f993528f640 1 -- 172.21.15.29:0/1153683665 shutdown_connections 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.430+0000 7f993528f640 1 -- 172.21.15.29:0/1153683665 wait complete. 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.430+0000 7f993528f640 10 monclient: build_initial_monmap 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 10 monclient: monmap: 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:26.433 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 librados: starting msgr at 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 librados: starting objecter 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 Processor -- start 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 -- start start 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 librados: setting wanted keys 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 1 librados: calling monclient init 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.431+0000 7f993528f640 10 monclient: init 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:26.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient(hunting): picked mon.noname-a con 0x7f99300705b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient(hunting): picked mon.noname-b con 0x7f99301402d0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient(hunting): picked mon.noname-c con 0x7f9930149ae0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9930137fd0 con 0x7f99300705b0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993013fbd0 con 0x7f99301402d0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993013feb0 con 0x7f9930149ae0 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:26.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.432+0000 7f993528f640 10 monclient(hunting): authenticate will time out at 6563.185479s 2024-03-25T19:41:26.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.447+0000 7f992f7fe640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f9930149ae0 0x7f99301e1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:44964/0 (socket says 172.21.15.29:44964) 2024-03-25T19:41:26.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.447+0000 7f992f7fe640 1 -- 172.21.15.29:0/1294983923 learned_addr learned my addr 172.21.15.29:0/1294983923 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:26.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.450+0000 7f99177fe640 10 client.?.objecter ms_handle_connect 0x7f9930149ae0 2024-03-25T19:41:26.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.450+0000 7f99177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.451+0000 7f99177fe640 10 client.?.objecter ms_handle_connect 0x7f99301402d0 2024-03-25T19:41:26.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.451+0000 7f99177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.452 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.453+0000 7f99177fe640 10 client.?.objecter ms_handle_connect 0x7f99300705b0 2024-03-25T19:41:26.452 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.453+0000 7f99177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711284508 0 0) 0x7f993013fbd0 con 0x7f99301402d0 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:41:26.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.465+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c004eb0 con 0x7f99301402d0 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777881720 0 0) 0x7f9930137fd0 con 0x7f99300705b0 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:41:26.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.466+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993013fbd0 con 0x7f99300705b0 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085617759 0 0) 0x7f993013feb0 con 0x7f9930149ae0 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:41:26.470 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.470+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930137fd0 con 0x7f9930149ae0 2024-03-25T19:41:26.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.481+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3922599821 0 0) 0x7f991c004eb0 con 0x7f99301402d0 2024-03-25T19:41:26.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.481+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993013feb0 con 0x7f99301402d0 2024-03-25T19:41:26.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.481+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f99240018b0 con 0x7f99301402d0 2024-03-25T19:41:26.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.482+0000 7f99177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.482+0000 7f99177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.482+0000 7f99177fe640 10 monclient(hunting): dump: 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.482+0000 7f99177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2434624152 0 0) 0x7f993013fbd0 con 0x7f99300705b0 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f991c004eb0 con 0x7f99300705b0 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9920001870 con 0x7f99300705b0 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.485+0000 7f99177fe640 10 monclient(hunting): dump: 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.486 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.487 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.487 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:26.487 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:26.487 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:26.487 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 535149285 0 0) 0x7f991c004eb0 con 0x7f99300705b0 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 >> v1:172.21.15.29:6790/0 conn(0x7f9930149ae0 legacy=0x7f99301e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 >> v1:172.21.15.196:6789/0 conn(0x7f99301402d0 legacy=0x7f993013e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 1 monclient: found mon.a 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:26.493 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993013eff0 con 0x7f99300705b0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f99177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 10 monclient: _renew_subs 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9930137bb0 con 0x7f99300705b0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 10 monclient: _renew_subs 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.493+0000 7f993528f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.494+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99301490c0 con 0x7f99300705b0 2024-03-25T19:41:26.494 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.494+0000 7f993528f640 1 librados: init done 2024-03-25T19:41:26.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.494+0000 7f993528f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.495+0000 7f993528f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.495+0000 7f993528f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:26.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.495+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99300627b0 con 0x7f99300705b0 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.807+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9920003610 con 0x7f99300705b0 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9920001d80 con 0x7f99300705b0 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 10 monclient: dump: 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:26.808 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f992d7fa640 4 set_mon_vals no callback set 2024-03-25T19:41:26.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.808+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9920003b20 con 0x7f99300705b0 2024-03-25T19:41:26.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.809+0000 7f992e7fc640 10 monclient: get_auth_request con 0x7f991c06dfb0 auth_method 0 2024-03-25T19:41:26.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.809+0000 7f99177fe640 10 client.4566.objecter ms_handle_connect 0x7f991c06dfb0 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f99200535c0 con 0x7f99300705b0 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 10 client.4566.objecter ms_dispatch 0x7f9930000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f992004f0d0 con 0x7f99300705b0 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.816 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:26.816+0000 7f99177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:27.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.079+0000 7f993528f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:27.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.079+0000 7f993528f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:27.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.079+0000 7f993528f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:27.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.079+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f993013f180 con 0x7f99300705b0 2024-03-25T19:41:27.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.117+0000 7f99177fe640 1 -- 172.21.15.29:0/1294983923 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f9920051000 con 0x7f99300705b0 2024-03-25T19:41:27.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.117+0000 7f99177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:27.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.117+0000 7f99177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:27.117 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:27.117 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.119+0000 7f993528f640 10 librados: watch_flush enter 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 10 librados: watch_flush exit 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 10 client.4566.objecter successfully canceled tick 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 >> v1:172.21.15.29:6800/2314944989 conn(0x7f991c06dfb0 legacy=0x7f991c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 10 monclient: shutdown 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 >> v1:172.21.15.29:6789/0 conn(0x7f99300705b0 legacy=0x7f993006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 shutdown_connections 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 >> 172.21.15.29:0/1294983923 conn(0x7f99300b5d10 msgr2=0x7f9930140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:27.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 shutdown_connections 2024-03-25T19:41:27.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 -- 172.21.15.29:0/1294983923 wait complete. 2024-03-25T19:41:27.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:27.120+0000 7f993528f640 1 librados: shutdown 2024-03-25T19:41:27.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:30.884 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:01.047 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.938 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:01.101 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.963 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:01.126 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.987 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:01.149 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.987 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:01.151 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.993 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:01.156 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:30.994 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:01.158 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:31.025 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:01.187 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:31.041 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:01.204 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:33.938 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:34.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.044+0000 7fab9288f640 10 monclient: get_monmap_and_config 2024-03-25T19:41:34.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.044+0000 7fab9288f640 10 monclient: build_initial_monmap 2024-03-25T19:41:34.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.045+0000 7fab9288f640 10 monclient: monmap: 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:34.045796+0000 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:34.045796+0000 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:34.045 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.045+0000 7fab9288f640 1 Processor -- start 2024-03-25T19:41:34.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.045+0000 7fab9288f640 1 -- start start 2024-03-25T19:41:34.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.045+0000 7fab9288f640 10 monclient: init 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-a con 0x7fab8c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-c con 0x7fab8c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-b con 0x7fab8c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c142520 con 0x7fab8c0a98f0 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0b28a0 con 0x7fab8c13f710 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0b2a30 con 0x7fab8c1430c0 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab9288f640 10 monclient(hunting): authenticate will time out at 6570.799237s 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab8b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fab8c13f710 0x7fab8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41560/0 (socket says 172.21.15.29:41560) 2024-03-25T19:41:34.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.046+0000 7fab8b7fe640 1 -- 172.21.15.29:0/3677744066 learned_addr learned my addr 172.21.15.29:0/3677744066 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294540391 0 0) 0x7fab8c142520 con 0x7fab8c0a98f0 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:41:34.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.094+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74003520 con 0x7fab8c0a98f0 2024-03-25T19:41:34.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456609358 0 0) 0x7fab74003520 con 0x7fab8c0a98f0 2024-03-25T19:41:34.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 >> v1:172.21.15.29:6790/0 conn(0x7fab8c1430c0 legacy=0x7fab8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 >> v1:172.21.15.196:6789/0 conn(0x7fab8c13f710 legacy=0x7fab8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:34.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 monclient: found mon.noname-a 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c0b44c0 con 0x7fab8c0a98f0 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: _finish_auth 0 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fab7c001690 con 0x7fab8c0a98f0 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab9288f640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab8affd640 10 monclient: dump: 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:34.106 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.105+0000 7fab9288f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab7c0017f0 con 0x7fab8c0a98f0 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 1 -- 172.21.15.29:0/3677744066 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fab7c001c00 con 0x7fab8c0a98f0 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab9288f640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab9288f640 4 set_mon_vals no callback set 2024-03-25T19:41:34.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.129+0000 7fab8affd640 10 monclient: dump: 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:34.130 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 10 monclient: shutdown 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 1 -- 172.21.15.29:0/3677744066 >> v1:172.21.15.29:6789/0 conn(0x7fab8c0a98f0 legacy=0x7fab8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 1 -- 172.21.15.29:0/3677744066 shutdown_connections 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 1 -- 172.21.15.29:0/3677744066 >> 172.21.15.29:0/3677744066 conn(0x7fab8c0b7ea0 msgr2=0x7fab8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 1 -- 172.21.15.29:0/3677744066 shutdown_connections 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.130+0000 7fab9288f640 1 -- 172.21.15.29:0/3677744066 wait complete. 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.131+0000 7fab9288f640 10 monclient: build_initial_monmap 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.131+0000 7fab9288f640 10 monclient: monmap: 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:34.132 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 librados: starting msgr at 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 librados: starting objecter 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 Processor -- start 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 -- start start 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 librados: setting wanted keys 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 1 librados: calling monclient init 2024-03-25T19:41:34.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.132+0000 7fab9288f640 10 monclient: init 2024-03-25T19:41:34.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:34.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:34.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-c con 0x7fab8c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-a con 0x7fab8c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient(hunting): picked mon.noname-b con 0x7fab8c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0abd30 con 0x7fab8c13f710 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c1d77a0 con 0x7fab8c1430c0 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c1d51f0 con 0x7fab8c0a98f0 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:34.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.133+0000 7fab9288f640 10 monclient(hunting): authenticate will time out at 6570.886274s 2024-03-25T19:41:34.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.148+0000 7fab90e2c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fab8c1430c0 0x7fab8c1e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:41604/0 (socket says 172.21.15.29:41604) 2024-03-25T19:41:34.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.148+0000 7fab90e2c640 1 -- 172.21.15.29:0/1058875289 learned_addr learned my addr 172.21.15.29:0/1058875289 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:34.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.149+0000 7fab88ff9640 10 client.?.objecter ms_handle_connect 0x7fab8c1430c0 2024-03-25T19:41:34.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.149+0000 7fab88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:34.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.152+0000 7fab88ff9640 10 client.?.objecter ms_handle_connect 0x7fab8c13f710 2024-03-25T19:41:34.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.152+0000 7fab88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:34.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.162+0000 7fab88ff9640 10 client.?.objecter ms_handle_connect 0x7fab8c0a98f0 2024-03-25T19:41:34.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.162+0000 7fab88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:34.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021990193 0 0) 0x7fab8c0abd30 con 0x7fab8c13f710 2024-03-25T19:41:34.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:34.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:34.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:34.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:41:34.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.167+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78004eb0 con 0x7fab8c13f710 2024-03-25T19:41:34.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964615473 0 0) 0x7fab8c1d77a0 con 0x7fab8c1430c0 2024-03-25T19:41:34.239 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:34.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:34.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:34.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:41:34.240 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.240+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c0abd30 con 0x7fab8c1430c0 2024-03-25T19:41:34.241 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851756426 0 0) 0x7fab78004eb0 con 0x7fab8c13f710 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab8c1d77a0 con 0x7fab8c13f710 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fab800018a0 con 0x7fab8c13f710 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:34.242 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:34.243 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:34.243 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.243 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.242+0000 7fab88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 589211589 0 0) 0x7fab8c1d77a0 con 0x7fab8c13f710 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 >> v1:172.21.15.29:6790/0 conn(0x7fab8c0a98f0 legacy=0x7fab8c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 >> v1:172.21.15.196:6789/0 conn(0x7fab8c1430c0 legacy=0x7fab8c1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 1 monclient: found mon.a 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c1d83d0 con 0x7fab8c13f710 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:34.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 5 monclient: authenticate success, global_id 4572 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 10 monclient: _renew_subs 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c138a50 con 0x7fab8c13f710 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 10 monclient: _renew_subs 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c1e35c0 con 0x7fab8c13f710 2024-03-25T19:41:34.250 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.249+0000 7fab9288f640 1 librados: init done 2024-03-25T19:41:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.250+0000 7fab9288f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.250+0000 7fab9288f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.250+0000 7fab9288f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.251 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.250+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c1d7990 con 0x7fab8c13f710 2024-03-25T19:41:34.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab80001d10 con 0x7fab8c13f710 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fab80004a40 con 0x7fab8c13f710 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab88ff9640 10 monclient: dump: 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:34.599 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:34.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.598+0000 7fab8a7fc640 4 set_mon_vals no callback set 2024-03-25T19:41:34.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.599+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fab80004c60 con 0x7fab8c13f710 2024-03-25T19:41:34.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.600+0000 7fab8bfff640 10 monclient: get_auth_request con 0x7fab7806dd40 auth_method 0 2024-03-25T19:41:34.600 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.600+0000 7fab88ff9640 10 client.4572.objecter ms_handle_connect 0x7fab7806dd40 2024-03-25T19:41:34.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fab80054740 con 0x7fab8c13f710 2024-03-25T19:41:34.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 10 client.4572.objecter ms_dispatch 0x7fab8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:34.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:34.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:34.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-25T19:41:34.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab80050250 con 0x7fab8c13f710 2024-03-25T19:41:34.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:34.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.606+0000 7fab88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:34.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.867+0000 7fab9288f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.867+0000 7fab9288f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.867+0000 7fab9288f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:41:34.867 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.867+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab8c0627b0 con 0x7fab8c13f710 2024-03-25T19:41:34.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.916+0000 7fab88ff9640 1 -- 172.21.15.29:0/1058875289 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fab80052180 con 0x7fab8c13f710 2024-03-25T19:41:34.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.916+0000 7fab88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.916+0000 7fab88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:34.916 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:34.917 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:34.918 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 10 librados: watch_flush enter 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 10 librados: watch_flush exit 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 10 client.4572.objecter successfully canceled tick 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 >> v1:172.21.15.29:6800/2314944989 conn(0x7fab7806dd40 legacy=0x7fab78070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 10 monclient: shutdown 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 >> v1:172.21.15.29:6789/0 conn(0x7fab8c13f710 legacy=0x7fab8c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 shutdown_connections 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 >> 172.21.15.29:0/1058875289 conn(0x7fab8c0b7ea0 msgr2=0x7fab8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.918+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 shutdown_connections 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.919+0000 7fab9288f640 1 -- 172.21.15.29:0/1058875289 wait complete. 2024-03-25T19:41:34.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:34.919+0000 7fab9288f640 1 librados: shutdown 2024-03-25T19:41:34.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:41.734 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.836+0000 7f0306ac1640 10 monclient: get_monmap_and_config 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.836+0000 7f0306ac1640 10 monclient: build_initial_monmap 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.836+0000 7f0306ac1640 10 monclient: monmap: 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:41.836 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:41.837677+0000 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:41.837677+0000 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.837+0000 7f0306ac1640 1 Processor -- start 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 1 -- start start 2024-03-25T19:41:41.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient: init 2024-03-25T19:41:41.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:41.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f03000a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f030013f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f03001430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0300142520 con 0x7f03000a98f0 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000b28a0 con 0x7f03001430c0 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000b2a30 con 0x7f030013f710 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:41.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.838+0000 7f0306ac1640 10 monclient(hunting): authenticate will time out at 6578.591706s 2024-03-25T19:41:41.854 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.854+0000 7f030485d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f03000a98f0 0x7f03000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42888/0 (socket says 172.21.15.29:42888) 2024-03-25T19:41:41.854 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.854+0000 7f030485d640 1 -- 172.21.15.29:0/3314471441 learned_addr learned my addr 172.21.15.29:0/3314471441 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:41.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334000828 0 0) 0x7f03000b28a0 con 0x7f03001430c0 2024-03-25T19:41:41.868 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:41:41.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.869+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f0003520 con 0x7f03001430c0 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769951363 0 0) 0x7f03000b2a30 con 0x7f030013f710 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:41:41.877 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.877+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000b28a0 con 0x7f030013f710 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025492212 0 0) 0x7f02f0003520 con 0x7f03001430c0 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 >> v1:172.21.15.29:6790/0 conn(0x7f030013f710 legacy=0x7f03001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 >> v1:172.21.15.29:6789/0 conn(0x7f03000a98f0 legacy=0x7f03000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000b44c0 con 0x7f03001430c0 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:41.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f02f8001a10 con 0x7f03001430c0 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f02ff7fe640 10 monclient: dump: 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:41.880 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:41.881 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f0306ac1640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:41:41.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.879+0000 7f0306ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:41.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.888+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f8001b70 con 0x7f03001430c0 2024-03-25T19:41:41.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.888+0000 7f02ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:41.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f02ff7fe640 1 -- 172.21.15.29:0/3314471441 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f02f8001f80 con 0x7f03001430c0 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 4 set_mon_vals no callback set 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f02ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f02ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f02ff7fe640 10 monclient: dump: 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:41.889 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:41.890 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 10 monclient: shutdown 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 1 -- 172.21.15.29:0/3314471441 >> v1:172.21.15.196:6789/0 conn(0x7f03001430c0 legacy=0x7f030014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 1 -- 172.21.15.29:0/3314471441 shutdown_connections 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 1 -- 172.21.15.29:0/3314471441 >> 172.21.15.29:0/3314471441 conn(0x7f03000b7ea0 msgr2=0x7f03000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 1 -- 172.21.15.29:0/3314471441 shutdown_connections 2024-03-25T19:41:41.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.889+0000 7f0306ac1640 1 -- 172.21.15.29:0/3314471441 wait complete. 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.890+0000 7f0306ac1640 10 monclient: build_initial_monmap 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.890+0000 7f0306ac1640 10 monclient: monmap: 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 librados: starting msgr at 2024-03-25T19:41:41.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 librados: starting objecter 2024-03-25T19:41:41.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 Processor -- start 2024-03-25T19:41:41.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 -- start start 2024-03-25T19:41:41.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 librados: setting wanted keys 2024-03-25T19:41:41.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 1 librados: calling monclient init 2024-03-25T19:41:41.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.891+0000 7f0306ac1640 10 monclient: init 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f03000a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f030013f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f03001430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03001369d0 con 0x7f03000a98f0 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0300133a60 con 0x7f03001430c0 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000ab9e0 con 0x7f030013f710 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:41.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.892+0000 7f0306ac1640 10 monclient(hunting): authenticate will time out at 6578.645417s 2024-03-25T19:41:41.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.895+0000 7f030505e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f03001430c0 0x7f0300133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:56870/0 (socket says 172.21.15.29:56870) 2024-03-25T19:41:41.895 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.895+0000 7f030505e640 1 -- 172.21.15.29:0/3804015007 learned_addr learned my addr 172.21.15.29:0/3804015007 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:41.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.898+0000 7f02fd7fa640 10 client.?.objecter ms_handle_connect 0x7f03001430c0 2024-03-25T19:41:41.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.898+0000 7f02fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26373403 0 0) 0x7f0300133a60 con 0x7f03001430c0 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:41:41.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.901+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e4004eb0 con 0x7f03001430c0 2024-03-25T19:41:41.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.904+0000 7f02fd7fa640 10 client.?.objecter ms_handle_connect 0x7f030013f710 2024-03-25T19:41:41.903 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.904+0000 7f02fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.911+0000 7f02fd7fa640 10 client.?.objecter ms_handle_connect 0x7f03000a98f0 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.911+0000 7f02fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755728674 0 0) 0x7f03000ab9e0 con 0x7f030013f710 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0300133a60 con 0x7f030013f710 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2291456354 0 0) 0x7f02e4004eb0 con 0x7f03001430c0 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03000ab9e0 con 0x7f03001430c0 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f02f8001f80 con 0x7f03001430c0 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:41.921 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.921+0000 7f02fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:41.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.928+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954603615 0 0) 0x7f03001369d0 con 0x7f03000a98f0 2024-03-25T19:41:41.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.929+0000 7f02fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.929+0000 7f02fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.929+0000 7f02fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.929+0000 7f02fd7fa640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:41:41.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.929+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e4004eb0 con 0x7f03000a98f0 2024-03-25T19:41:41.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4219398653 0 0) 0x7f0300133a60 con 0x7f030013f710 2024-03-25T19:41:41.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03001369d0 con 0x7f030013f710 2024-03-25T19:41:41.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f02ec001930 con 0x7f030013f710 2024-03-25T19:41:41.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.930+0000 7f02fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:41.930 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1846368676 0 0) 0x7f03000ab9e0 con 0x7f03001430c0 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 >> v1:172.21.15.29:6790/0 conn(0x7f030013f710 legacy=0x7f03001362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 >> v1:172.21.15.29:6789/0 conn(0x7f03000a98f0 legacy=0x7f0300132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 1 monclient: found mon.b 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000abcc0 con 0x7f03001430c0 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f02fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:41:41.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 10 monclient: _renew_subs 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03000abfa0 con 0x7f03001430c0 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 10 monclient: _renew_subs 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0300137e40 con 0x7f03001430c0 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.946+0000 7f0306ac1640 1 librados: init done 2024-03-25T19:41:41.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.947+0000 7f0306ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:41.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.947+0000 7f0306ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:41.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.947+0000 7f0306ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:41.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:41.947+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03000627b0 con 0x7f03001430c0 2024-03-25T19:41:42.218 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f8003c20 con 0x7f03001430c0 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f02f8003ff0 con 0x7f03001430c0 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02fd7fa640 10 monclient: dump: 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:42.219 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:42.220 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:42.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.219+0000 7f02feffd640 4 set_mon_vals no callback set 2024-03-25T19:41:42.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.220+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f02f8004210 con 0x7f03001430c0 2024-03-25T19:41:42.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.220+0000 7f030485d640 10 monclient: get_auth_request con 0x7f02e406dfb0 auth_method 0 2024-03-25T19:41:42.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.221+0000 7f02fd7fa640 10 client.4579.objecter ms_handle_connect 0x7f02e406dfb0 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f02f8054bc0 con 0x7f03001430c0 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 10 client.4579.objecter ms_dispatch 0x7f0300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02f80506d0 con 0x7f03001430c0 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:42.225 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.225+0000 7f02fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:42.481 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.482+0000 7f0306ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.482+0000 7f0306ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.482+0000 7f0306ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:42.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.482+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f030013dfd0 con 0x7f03001430c0 2024-03-25T19:41:42.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.505+0000 7f02fd7fa640 1 -- 172.21.15.29:0/3804015007 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f02f8052600 con 0x7f03001430c0 2024-03-25T19:41:42.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.505+0000 7f02fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.505+0000 7f02fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:42.505 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:42.505 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 10 librados: watch_flush enter 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 10 librados: watch_flush exit 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 10 client.4579.objecter successfully canceled tick 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 >> v1:172.21.15.29:6800/2314944989 conn(0x7f02e406dfb0 legacy=0x7f02e4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.508 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 10 monclient: shutdown 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 >> v1:172.21.15.196:6789/0 conn(0x7f03001430c0 legacy=0x7f0300133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 shutdown_connections 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 >> 172.21.15.29:0/3804015007 conn(0x7f03000b7ea0 msgr2=0x7f0300074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 shutdown_connections 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 -- 172.21.15.29:0/3804015007 wait complete. 2024-03-25T19:41:42.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:42.508+0000 7f0306ac1640 1 librados: shutdown 2024-03-25T19:41:42.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:49.328 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.438+0000 7ff43b04a640 10 monclient: get_monmap_and_config 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.438+0000 7ff43b04a640 10 monclient: build_initial_monmap 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.438+0000 7ff43b04a640 10 monclient: monmap: 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:49.439136+0000 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:49.439136+0000 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:49.438 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.439 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:49.439 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:49.439 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:49.439 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.439+0000 7ff43b04a640 1 Processor -- start 2024-03-25T19:41:49.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.439+0000 7ff43b04a640 1 -- start start 2024-03-25T19:41:49.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.439+0000 7ff43b04a640 10 monclient: init 2024-03-25T19:41:49.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:49.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-b con 0x7ff4340b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-c con 0x7ff4340b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-a con 0x7ff43414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4340af940 con 0x7ff43414be90 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4340b0a70 con 0x7ff4340b1730 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4340b0d50 con 0x7ff4340b4d60 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:49.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.440+0000 7ff43b04a640 10 monclient(hunting): authenticate will time out at 6586.193592s 2024-03-25T19:41:49.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.453+0000 7ff433fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff4340b1730 0x7ff4340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50066/0 (socket says 172.21.15.29:50066) 2024-03-25T19:41:49.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.453+0000 7ff433fff640 1 -- 172.21.15.29:0/3510218923 learned_addr learned my addr 172.21.15.29:0/3510218923 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730929545 0 0) 0x7ff4340b0a70 con 0x7ff4340b1730 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:41:49.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.463+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424003520 con 0x7ff4340b1730 2024-03-25T19:41:49.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596632586 0 0) 0x7ff4340af940 con 0x7ff43414be90 2024-03-25T19:41:49.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.464+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4340b0a70 con 0x7ff43414be90 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091477746 0 0) 0x7ff4340b0d50 con 0x7ff4340b4d60 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:41:49.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.465+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4340af940 con 0x7ff4340b4d60 2024-03-25T19:41:49.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560250904 0 0) 0x7ff424003520 con 0x7ff4340b1730 2024-03-25T19:41:49.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 >> v1:172.21.15.29:6790/0 conn(0x7ff4340b4d60 legacy=0x7ff43413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 >> v1:172.21.15.29:6789/0 conn(0x7ff43414be90 legacy=0x7ff43414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340b1030 con 0x7ff4340b1730 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff420001690 con 0x7ff4340b1730 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff4337fe640 10 monclient: dump: 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.476 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff43b04a640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:41:49.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.475+0000 7ff43b04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4200018b0 con 0x7ff4340b1730 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 1 -- 172.21.15.29:0/3510218923 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff420001c40 con 0x7ff4340b1730 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff43b04a640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff43b04a640 4 set_mon_vals no callback set 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.507+0000 7ff4337fe640 10 monclient: dump: 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.507 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.508 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:49.508 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:49.508 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:49.508 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 10 monclient: shutdown 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 1 -- 172.21.15.29:0/3510218923 >> v1:172.21.15.196:6789/0 conn(0x7ff4340b1730 legacy=0x7ff4340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 1 -- 172.21.15.29:0/3510218923 shutdown_connections 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 1 -- 172.21.15.29:0/3510218923 >> 172.21.15.29:0/3510218923 conn(0x7ff4340617a0 msgr2=0x7ff434061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:49.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 1 -- 172.21.15.29:0/3510218923 shutdown_connections 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.508+0000 7ff43b04a640 1 -- 172.21.15.29:0/3510218923 wait complete. 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.509+0000 7ff43b04a640 10 monclient: build_initial_monmap 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.509+0000 7ff43b04a640 10 monclient: monmap: 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:49.510 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.509+0000 7ff43b04a640 1 librados: starting msgr at 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.509+0000 7ff43b04a640 1 librados: starting objecter 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 1 Processor -- start 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 1 -- start start 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 1 librados: setting wanted keys 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 1 librados: calling monclient init 2024-03-25T19:41:49.511 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient: init 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-c con 0x7ff4340b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-b con 0x7ff4340b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.510+0000 7ff43b04a640 10 monclient(hunting): picked mon.noname-a con 0x7ff43414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.511+0000 7ff43b04a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341d78e0 con 0x7ff43414be90 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.511+0000 7ff43b04a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341d7bc0 con 0x7ff4340b4d60 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.511+0000 7ff43b04a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341d8760 con 0x7ff4340b1730 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.511+0000 7ff43b04a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:49.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.511+0000 7ff43b04a640 10 monclient(hunting): authenticate will time out at 6586.264011s 2024-03-25T19:41:49.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.519+0000 7ff438de6640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7ff4340b1730 0x7ff434142850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:47728/0 (socket says 172.21.15.29:47728) 2024-03-25T19:41:49.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.519+0000 7ff438de6640 1 -- 172.21.15.29:0/3720438776 learned_addr learned my addr 172.21.15.29:0/3720438776 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:49.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.523+0000 7ff4317fa640 10 client.?.objecter ms_handle_connect 0x7ff4340b1730 2024-03-25T19:41:49.522 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.523+0000 7ff4317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.524+0000 7ff4317fa640 10 client.?.objecter ms_handle_connect 0x7ff4340b4d60 2024-03-25T19:41:49.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.524+0000 7ff4317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.531+0000 7ff4317fa640 10 client.?.objecter ms_handle_connect 0x7ff43414be90 2024-03-25T19:41:49.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.531+0000 7ff4317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412068318 0 0) 0x7ff4341d7bc0 con 0x7ff4340b4d60 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:41:49.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.532+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff418004eb0 con 0x7ff4340b4d60 2024-03-25T19:41:49.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884943434 0 0) 0x7ff4341d78e0 con 0x7ff43414be90 2024-03-25T19:41:49.543 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.543 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.543 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.543 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:41:49.543 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.543+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4341d7bc0 con 0x7ff43414be90 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527539713 0 0) 0x7ff4341d8760 con 0x7ff4340b1730 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:41:49.549 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.549+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4341d78e0 con 0x7ff4340b1730 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1100736146 0 0) 0x7ff418004eb0 con 0x7ff4340b4d60 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4341d8760 con 0x7ff4340b4d60 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff4200015d0 con 0x7ff4340b4d60 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 10 monclient(hunting): dump: 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.553 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.552+0000 7ff4317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:49.571 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.565+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322706623 0 0) 0x7ff4341d78e0 con 0x7ff4340b1730 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff418004eb0 con 0x7ff4340b1730 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff4280019b0 con 0x7ff4340b1730 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient(hunting): dump: 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:49.572 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 452283951 0 0) 0x7ff4341d8760 con 0x7ff4340b4d60 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 >> v1:172.21.15.29:6790/0 conn(0x7ff4340b1730 legacy=0x7ff434142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 >> v1:172.21.15.29:6789/0 conn(0x7ff43414be90 legacy=0x7ff434137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 monclient: found mon.b 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340a9e20 con 0x7ff4340b4d60 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff4317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 10 monclient: _renew_subs 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4340a96e0 con 0x7ff4340b4d60 2024-03-25T19:41:49.573 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 10 monclient: _renew_subs 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff434140380 con 0x7ff4340b4d60 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.566+0000 7ff43b04a640 1 librados: init done 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.567+0000 7ff43b04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.567+0000 7ff43b04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.567+0000 7ff43b04a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:49.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.567+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4340aa030 con 0x7ff4340b4d60 2024-03-25T19:41:49.847 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff420004a90 con 0x7ff4340b4d60 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff420004da0 con 0x7ff4340b4d60 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff4317fa640 10 monclient: dump: 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:49.848 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.848+0000 7ff432ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.849+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ff42001ad70 con 0x7ff4340b4d60 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.850+0000 7ff438de6640 10 monclient: get_auth_request con 0x7ff41806df20 auth_method 0 2024-03-25T19:41:49.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.850+0000 7ff4317fa640 10 client.4585.objecter ms_handle_connect 0x7ff41806df20 2024-03-25T19:41:49.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.851+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ff420054880 con 0x7ff4340b4d60 2024-03-25T19:41:49.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.851+0000 7ff4317fa640 10 client.4585.objecter ms_dispatch 0x7ff434000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:49.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.851+0000 7ff4317fa640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:49.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.851+0000 7ff4317fa640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:49.851 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.851+0000 7ff4317fa640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:41:49.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.854+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff420050390 con 0x7ff4340b4d60 2024-03-25T19:41:49.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.854+0000 7ff4317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:49.854+0000 7ff4317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:50.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.118+0000 7ff43b04a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.118+0000 7ff43b04a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.118+0000 7ff43b04a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:50.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.118+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4341401f0 con 0x7ff4340b4d60 2024-03-25T19:41:50.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.149+0000 7ff4317fa640 1 -- 172.21.15.29:0/3720438776 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ff4200522c0 con 0x7ff4340b4d60 2024-03-25T19:41:50.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.149+0000 7ff4317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.149+0000 7ff4317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:50.149 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:50.149 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:50.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 10 librados: watch_flush enter 2024-03-25T19:41:50.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 10 librados: watch_flush exit 2024-03-25T19:41:50.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 >> v1:172.21.15.29:6800/2314944989 conn(0x7ff41806df20 legacy=0x7ff4180703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 10 monclient: shutdown 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 >> v1:172.21.15.196:6789/0 conn(0x7ff4340b4d60 legacy=0x7ff4341d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 shutdown_connections 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 >> 172.21.15.29:0/3720438776 conn(0x7ff4340617a0 msgr2=0x7ff434061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 shutdown_connections 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.151+0000 7ff43b04a640 1 -- 172.21.15.29:0/3720438776 wait complete. 2024-03-25T19:41:50.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:50.152+0000 7ff43b04a640 1 librados: shutdown 2024-03-25T19:41:50.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:51.119 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:21.283 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.174 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:21.337 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.211 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:21.374 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.233 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:21.396 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.237 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:21.400 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.243 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:21.405 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.251 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:21.414 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.277 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:21.439 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:51.281 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:21.444 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.968 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.082+0000 7f262ba4a640 10 monclient: get_monmap_and_config 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.082+0000 7f262ba4a640 10 monclient: build_initial_monmap 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.082+0000 7f262ba4a640 10 monclient: monmap: 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.082 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:41:57.083445+0000 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:41:57.083445+0000 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:41:57.083 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.083+0000 7f262ba4a640 1 Processor -- start 2024-03-25T19:41:57.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.083+0000 7f262ba4a640 1 -- start start 2024-03-25T19:41:57.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.083+0000 7f262ba4a640 10 monclient: init 2024-03-25T19:41:57.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-c con 0x7f26240a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-a con 0x7f262413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-b con 0x7f26241430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2624142520 con 0x7f262413f710 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240b28a0 con 0x7f26240a98f0 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240b2a30 con 0x7f26241430c0 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:57.085 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.084+0000 7f262ba4a640 10 monclient(hunting): authenticate will time out at 6593.837703s 2024-03-25T19:41:57.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.086+0000 7f26297e6640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f26240a98f0 0x7f26240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:54458/0 (socket says 172.21.15.29:54458) 2024-03-25T19:41:57.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.086+0000 7f26297e6640 1 -- 172.21.15.29:0/1489664834 learned_addr learned my addr 172.21.15.29:0/1489664834 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091470271 0 0) 0x7f26240b28a0 con 0x7f26240a98f0 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:41:57.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.097+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610003520 con 0x7f26240a98f0 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327492724 0 0) 0x7f2610003520 con 0x7f26240a98f0 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 >> v1:172.21.15.29:6790/0 conn(0x7f26241430c0 legacy=0x7f262414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 >> v1:172.21.15.29:6789/0 conn(0x7f262413f710 legacy=0x7f26241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 monclient: found mon.noname-c 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240b44c0 con 0x7f26240a98f0 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2620001690 con 0x7f26240a98f0 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f261bfff640 10 monclient: dump: 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.109 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f262ba4a640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:41:57.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.108+0000 7f262ba4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:57.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2620001920 con 0x7f26240a98f0 2024-03-25T19:41:57.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:57.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 1 -- 172.21.15.29:0/1489664834 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f2620001cb0 con 0x7f26240a98f0 2024-03-25T19:41:57.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 4 set_mon_vals no callback set 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f261bfff640 10 monclient: dump: 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.117 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 10 monclient: shutdown 2024-03-25T19:41:57.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 1 -- 172.21.15.29:0/1489664834 >> v1:172.21.15.196:6789/0 conn(0x7f26240a98f0 legacy=0x7f26240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 1 -- 172.21.15.29:0/1489664834 shutdown_connections 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.117+0000 7f262ba4a640 1 -- 172.21.15.29:0/1489664834 >> 172.21.15.29:0/1489664834 conn(0x7f26240b7ea0 msgr2=0x7f26240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.118+0000 7f262ba4a640 1 -- 172.21.15.29:0/1489664834 shutdown_connections 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.118+0000 7f262ba4a640 1 -- 172.21.15.29:0/1489664834 wait complete. 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.118+0000 7f262ba4a640 10 monclient: build_initial_monmap 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.118+0000 7f262ba4a640 10 monclient: monmap: 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:41:57.120 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 librados: starting msgr at 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 librados: starting objecter 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 Processor -- start 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 -- start start 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 librados: setting wanted keys 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 librados: calling monclient init 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient: init 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-c con 0x7f26240a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-a con 0x7f262413f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient(hunting): picked mon.noname-b con 0x7f26241430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2624138cb0 con 0x7f262413f710 2024-03-25T19:41:57.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2624137030 con 0x7f26241430c0 2024-03-25T19:41:57.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26241372d0 con 0x7f26240a98f0 2024-03-25T19:41:57.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:57.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.119+0000 7f262ba4a640 10 monclient(hunting): authenticate will time out at 6593.872816s 2024-03-25T19:41:57.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.122+0000 7f2629fe7640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f26241430c0 0x7f2624139420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:54484/0 (socket says 172.21.15.29:54484) 2024-03-25T19:41:57.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.122+0000 7f2629fe7640 1 -- 172.21.15.29:0/646666315 learned_addr learned my addr 172.21.15.29:0/646666315 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:41:57.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.124+0000 7f2619ffb640 10 client.?.objecter ms_handle_connect 0x7f26241430c0 2024-03-25T19:41:57.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.124+0000 7f2619ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.128+0000 7f2619ffb640 10 client.?.objecter ms_handle_connect 0x7f262413f710 2024-03-25T19:41:57.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.128+0000 7f2619ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.132+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976922417 0 0) 0x7f2624138cb0 con 0x7f262413f710 2024-03-25T19:41:57.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2608004eb0 con 0x7f262413f710 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958061418 0 0) 0x7f2624137030 con 0x7f26241430c0 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:41:57.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.133+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624138cb0 con 0x7f26241430c0 2024-03-25T19:41:57.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.140+0000 7f2619ffb640 10 client.?.objecter ms_handle_connect 0x7f26240a98f0 2024-03-25T19:41:57.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.140+0000 7f2619ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444574643 0 0) 0x7f26241372d0 con 0x7f26240a98f0 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:41:57.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.145+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624137030 con 0x7f26240a98f0 2024-03-25T19:41:57.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417681226 0 0) 0x7f2624138cb0 con 0x7f26241430c0 2024-03-25T19:41:57.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26241372d0 con 0x7f26241430c0 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f261c001c50 con 0x7f26241430c0 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.149 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.148+0000 7f2619ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:57.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840267788 0 0) 0x7f2608004eb0 con 0x7f262413f710 2024-03-25T19:41:57.151 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2624138cb0 con 0x7f262413f710 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f26140018c0 con 0x7f262413f710 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.152+0000 7f2619ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.152 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786254827 0 0) 0x7f2624137030 con 0x7f26240a98f0 2024-03-25T19:41:57.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2608004eb0 con 0x7f26240a98f0 2024-03-25T19:41:57.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f26200014b0 con 0x7f26240a98f0 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.159+0000 7f2619ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.159 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1680488397 0 0) 0x7f26241372d0 con 0x7f26241430c0 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 >> v1:172.21.15.29:6790/0 conn(0x7f26240a98f0 legacy=0x7f26241d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 >> v1:172.21.15.29:6789/0 conn(0x7f262413f710 legacy=0x7f26241377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 1 monclient: found mon.b 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:57.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240b7090 con 0x7f26241430c0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f2619ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 10 monclient: _renew_subs 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26240b7340 con 0x7f26241430c0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 10 monclient: _renew_subs 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26241330e0 con 0x7f26241430c0 2024-03-25T19:41:57.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.168+0000 7f262ba4a640 1 librados: init done 2024-03-25T19:41:57.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.170+0000 7f262ba4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.170+0000 7f262ba4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.170+0000 7f262ba4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.170+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262414f2d0 con 0x7f26241430c0 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f261c003990 con 0x7f26241430c0 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f261c003d60 con 0x7f26241430c0 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f2619ffb640 10 monclient: dump: 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.453 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:41:57.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.453+0000 7f261b7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:57.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.454+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f261c003f80 con 0x7f26241430c0 2024-03-25T19:41:57.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.454+0000 7f26297e6640 10 monclient: get_auth_request con 0x7f260806e0f0 auth_method 0 2024-03-25T19:41:57.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.455+0000 7f2619ffb640 10 client.4591.objecter ms_handle_connect 0x7f260806e0f0 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.459+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f261c054ab0 con 0x7f26241430c0 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.459+0000 7f2619ffb640 10 client.4591.objecter ms_dispatch 0x7f2624000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.459+0000 7f2619ffb640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.459+0000 7f2619ffb640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.459+0000 7f2619ffb640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T19:41:57.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.461+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f261c0505c0 con 0x7f26241430c0 2024-03-25T19:41:57.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.461+0000 7f2619ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.461+0000 7f2619ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:57.699 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.699+0000 7f262ba4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.699 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.699+0000 7f262ba4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.699 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.699+0000 7f262ba4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:41:57.699 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.699+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26240627b0 con 0x7f26241430c0 2024-03-25T19:41:57.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.769+0000 7f2619ffb640 1 -- 172.21.15.29:0/646666315 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f261c0524f0 con 0x7f26241430c0 2024-03-25T19:41:57.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.769+0000 7f2619ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.769 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.769+0000 7f2619ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:57.770 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:41:57.770 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.771+0000 7f262ba4a640 10 librados: watch_flush enter 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.771+0000 7f262ba4a640 10 librados: watch_flush exit 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.771+0000 7f262ba4a640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.771+0000 7f262ba4a640 10 client.4591.objecter successfully canceled tick 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 >> v1:172.21.15.29:6800/2314944989 conn(0x7f260806e0f0 legacy=0x7f26080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 10 monclient: shutdown 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 >> v1:172.21.15.196:6789/0 conn(0x7f26241430c0 legacy=0x7f2624139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 shutdown_connections 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 >> 172.21.15.29:0/646666315 conn(0x7f26240b7ea0 msgr2=0x7f262413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 shutdown_connections 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 -- 172.21.15.29:0/646666315 wait complete. 2024-03-25T19:41:57.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:41:57.772+0000 7f262ba4a640 1 librados: shutdown 2024-03-25T19:41:57.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:04.591 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.706+0000 7fa8395af640 10 monclient: get_monmap_and_config 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.706+0000 7fa8395af640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.706+0000 7fa8395af640 10 monclient: monmap: 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:04.707407+0000 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:04.707407+0000 2024-03-25T19:42:04.706 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.707 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:04.707 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:04.707 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:04.707 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:04.707 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.707+0000 7fa8395af640 1 Processor -- start 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.707+0000 7fa8395af640 1 -- start start 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.707+0000 7fa8395af640 10 monclient: init 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-b con 0x7fa8340b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-a con 0x7fa8340b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-c con 0x7fa83414be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:04.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8340af940 con 0x7fa8340b1730 2024-03-25T19:42:04.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8340b0a70 con 0x7fa83414be90 2024-03-25T19:42:04.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8340b0d50 con 0x7fa8340b4d60 2024-03-25T19:42:04.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.709 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.708+0000 7fa8395af640 10 monclient(hunting): authenticate will time out at 6601.461305s 2024-03-25T19:42:04.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.714+0000 7fa832ffd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fa8340b4d60 0x7fa83413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:51118/0 (socket says 172.21.15.29:51118) 2024-03-25T19:42:04.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.714+0000 7fa832ffd640 1 -- 172.21.15.29:0/4077133150 learned_addr learned my addr 172.21.15.29:0/4077133150 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611991914 0 0) 0x7fa8340b0d50 con 0x7fa8340b4d60 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:42:04.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.723+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa81c003520 con 0x7fa8340b4d60 2024-03-25T19:42:04.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717662026 0 0) 0x7fa8340b0a70 con 0x7fa83414be90 2024-03-25T19:42:04.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:42:04.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.734+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8340b0d50 con 0x7fa83414be90 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289813337 0 0) 0x7fa81c003520 con 0x7fa8340b4d60 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 >> v1:172.21.15.196:6789/0 conn(0x7fa83414be90 legacy=0x7fa83414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 >> v1:172.21.15.29:6789/0 conn(0x7fa8340b1730 legacy=0x7fa8340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 monclient: found mon.noname-b 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340b1030 con 0x7fa8340b4d60 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa824001690 con 0x7fa8340b4d60 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.740+0000 7fa831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.741+0000 7fa831ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.741+0000 7fa831ffb640 10 monclient: dump: 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.741+0000 7fa8395af640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:42:04.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.741+0000 7fa8395af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.752 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8240017f0 con 0x7fa8340b4d60 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 1 -- 172.21.15.29:0/4077133150 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa824001c00 con 0x7fa8340b4d60 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa831ffb640 10 monclient: dump: 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:04.753 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:04.754 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:04.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa8395af640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:04.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.753+0000 7fa8395af640 4 set_mon_vals no callback set 2024-03-25T19:42:04.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 10 monclient: shutdown 2024-03-25T19:42:04.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 1 -- 172.21.15.29:0/4077133150 >> v1:172.21.15.29:6790/0 conn(0x7fa8340b4d60 legacy=0x7fa83413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 1 -- 172.21.15.29:0/4077133150 shutdown_connections 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 1 -- 172.21.15.29:0/4077133150 >> 172.21.15.29:0/4077133150 conn(0x7fa8340617a0 msgr2=0x7fa834061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 1 -- 172.21.15.29:0/4077133150 shutdown_connections 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.754+0000 7fa8395af640 1 -- 172.21.15.29:0/4077133150 wait complete. 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 10 monclient: monmap: 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:04.756 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 librados: starting msgr at 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 librados: starting objecter 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 Processor -- start 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 -- start start 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 librados: setting wanted keys 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 1 librados: calling monclient init 2024-03-25T19:42:04.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.755+0000 7fa8395af640 10 monclient: init 2024-03-25T19:42:04.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:04.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-c con 0x7fa8340b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:04.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-b con 0x7fa8340b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:04.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient(hunting): picked mon.noname-a con 0x7fa83414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:04.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341db0c0 con 0x7fa83414be90 2024-03-25T19:42:04.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341dbc60 con 0x7fa8340b4d60 2024-03-25T19:42:04.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341dad50 con 0x7fa8340b1730 2024-03-25T19:42:04.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.756+0000 7fa8395af640 10 monclient(hunting): authenticate will time out at 6601.509829s 2024-03-25T19:42:04.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.763+0000 7fa832ffd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fa8340b1730 0x7fa8341db550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:51156/0 (socket says 172.21.15.29:51156) 2024-03-25T19:42:04.763 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.763+0000 7fa832ffd640 1 -- 172.21.15.29:0/795721869 learned_addr learned my addr 172.21.15.29:0/795721869 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:04.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.766+0000 7fa81b7fe640 10 client.?.objecter ms_handle_connect 0x7fa8340b1730 2024-03-25T19:42:04.766 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.766+0000 7fa81b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.768+0000 7fa81b7fe640 10 client.?.objecter ms_handle_connect 0x7fa83414be90 2024-03-25T19:42:04.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.768+0000 7fa81b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.773+0000 7fa81b7fe640 10 client.?.objecter ms_handle_connect 0x7fa8340b4d60 2024-03-25T19:42:04.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.773+0000 7fa81b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097525214 0 0) 0x7fa8341db0c0 con 0x7fa83414be90 2024-03-25T19:42:04.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:42:04.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.777+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820004eb0 con 0x7fa83414be90 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571110723 0 0) 0x7fa820004eb0 con 0x7fa83414be90 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8341db0c0 con 0x7fa83414be90 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa82c001bd0 con 0x7fa83414be90 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:04.787 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:04.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.787+0000 7fa81b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114832943 0 0) 0x7fa8341dbc60 con 0x7fa8340b4d60 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:42:04.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.789+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820004eb0 con 0x7fa8340b4d60 2024-03-25T19:42:04.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878859659 0 0) 0x7fa8341dad50 con 0x7fa8340b1730 2024-03-25T19:42:04.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:42:04.796 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.796+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8341dbc60 con 0x7fa8340b1730 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 925355727 0 0) 0x7fa8341db0c0 con 0x7fa83414be90 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 >> v1:172.21.15.29:6790/0 conn(0x7fa8340b1730 legacy=0x7fa8341db550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 >> v1:172.21.15.196:6789/0 conn(0x7fa8340b4d60 legacy=0x7fa834136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 1 monclient: found mon.a 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:04.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8341373a0 con 0x7fa83414be90 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa81b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 10 monclient: _renew_subs 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa834136540 con 0x7fa83414be90 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 10 monclient: _renew_subs 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa834136890 con 0x7fa83414be90 2024-03-25T19:42:04.802 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.801+0000 7fa8395af640 1 librados: init done 2024-03-25T19:42:04.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.802+0000 7fa8395af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.802+0000 7fa8395af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.802+0000 7fa8395af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:04.803 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:04.802+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83413bfa0 con 0x7fa83414be90 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa82c003990 con 0x7fa83414be90 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fa82c002100 con 0x7fa83414be90 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa81b7fe640 10 monclient: dump: 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:05.134 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:05.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.134+0000 7fa8317fa640 4 set_mon_vals no callback set 2024-03-25T19:42:05.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.135+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fa82c019b90 con 0x7fa83414be90 2024-03-25T19:42:05.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.136+0000 7fa832ffd640 10 monclient: get_auth_request con 0x7fa82006de10 auth_method 0 2024-03-25T19:42:05.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.136+0000 7fa81b7fe640 10 client.4596.objecter ms_handle_connect 0x7fa82006de10 2024-03-25T19:42:05.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.141+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fa82c053860 con 0x7fa83414be90 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.141+0000 7fa81b7fe640 10 client.4596.objecter ms_dispatch 0x7fa834000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.141+0000 7fa81b7fe640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.142+0000 7fa81b7fe640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.142+0000 7fa81b7fe640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.142+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa82c04f1a0 con 0x7fa83414be90 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.142+0000 7fa81b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.142+0000 7fa81b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:05.367 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.368+0000 7fa8395af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.368+0000 7fa8395af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.368+0000 7fa8395af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:05.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.368+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8341db270 con 0x7fa83414be90 2024-03-25T19:42:05.402 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.403+0000 7fa81b7fe640 1 -- 172.21.15.29:0/795721869 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fa8341db270 con 0x7fa83414be90 2024-03-25T19:42:05.402 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.403+0000 7fa81b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.402 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.403+0000 7fa81b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:05.403 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:05.403 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 10 librados: watch_flush enter 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 10 librados: watch_flush exit 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 10 client.4596.objecter successfully canceled tick 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 >> v1:172.21.15.29:6800/2314944989 conn(0x7fa82006de10 legacy=0x7fa8200702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 10 monclient: shutdown 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 >> v1:172.21.15.29:6789/0 conn(0x7fa83414be90 legacy=0x7fa83413b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 shutdown_connections 2024-03-25T19:42:05.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 >> 172.21.15.29:0/795721869 conn(0x7fa8340617a0 msgr2=0x7fa8340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 shutdown_connections 2024-03-25T19:42:05.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 -- 172.21.15.29:0/795721869 wait complete. 2024-03-25T19:42:05.407 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:05.406+0000 7fa8395af640 1 librados: shutdown 2024-03-25T19:42:05.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:11.029 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:41.192 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.075 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:41.238 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.125 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:41.288 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.152 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:41.314 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.156 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:41.319 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.156 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:41.319 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.162 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:41.325 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.189 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:08:41.351 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:11.195 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:41.358 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:12.225 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:12.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.325+0000 7f7ef171c640 10 monclient: get_monmap_and_config 2024-03-25T19:42:12.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.325+0000 7f7ef171c640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.325+0000 7f7ef171c640 10 monclient: monmap: 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:12.327574+0000 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:12.327574+0000 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:12.327 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.326+0000 7f7ef171c640 1 Processor -- start 2024-03-25T19:42:12.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.326+0000 7f7ef171c640 1 -- start start 2024-03-25T19:42:12.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.326+0000 7f7ef171c640 10 monclient: init 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-b con 0x7f7eec0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-a con 0x7f7eec13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-c con 0x7f7eec1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec142520 con 0x7f7eec13f710 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0b28a0 con 0x7f7eec1430c0 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0b2a30 con 0x7f7eec0a98f0 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.327+0000 7f7ef171c640 10 monclient(hunting): authenticate will time out at 6609.081269s 2024-03-25T19:42:12.332 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.331+0000 7f7eea7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7eec13f710 0x7f7eec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45314/0 (socket says 172.21.15.29:45314) 2024-03-25T19:42:12.332 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.331+0000 7f7eea7fc640 1 -- 172.21.15.29:0/1309513608 learned_addr learned my addr 172.21.15.29:0/1309513608 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552792729 0 0) 0x7f7eec142520 con 0x7f7eec13f710 2024-03-25T19:42:12.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed4003520 con 0x7f7eec13f710 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945232985 0 0) 0x7f7eec0b28a0 con 0x7f7eec1430c0 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.347 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:42:12.348 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.346+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec142520 con 0x7f7eec1430c0 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997668559 0 0) 0x7f7eec0b2a30 con 0x7f7eec0a98f0 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:42:12.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.358+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec0b28a0 con 0x7f7eec0a98f0 2024-03-25T19:42:12.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289807331 0 0) 0x7f7eec142520 con 0x7f7eec1430c0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 >> v1:172.21.15.29:6790/0 conn(0x7f7eec0a98f0 legacy=0x7f7eec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 >> v1:172.21.15.29:6789/0 conn(0x7f7eec13f710 legacy=0x7f7eec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0b44c0 con 0x7f7eec1430c0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.367+0000 7f7ee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ef171c640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f7ee40019f0 con 0x7f7eec1430c0 2024-03-25T19:42:12.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ef171c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ee9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ee9ffb640 10 monclient: dump: 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:12.370 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.368+0000 7f7ef171c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ee4001b50 con 0x7f7eec1430c0 2024-03-25T19:42:12.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:12.386 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 1 -- 172.21.15.29:0/1309513608 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f7ee4001f60 con 0x7f7eec1430c0 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ef171c640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ef171c640 4 set_mon_vals no callback set 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ee9ffb640 10 monclient: dump: 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:12.387 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:12.388 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:12.388 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ef171c640 10 monclient: shutdown 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ef171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.386+0000 7f7ef171c640 1 -- 172.21.15.29:0/1309513608 >> v1:172.21.15.196:6789/0 conn(0x7f7eec1430c0 legacy=0x7f7eec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 1 -- 172.21.15.29:0/1309513608 shutdown_connections 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 1 -- 172.21.15.29:0/1309513608 >> 172.21.15.29:0/1309513608 conn(0x7f7eec0b7ea0 msgr2=0x7f7eec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 1 -- 172.21.15.29:0/1309513608 shutdown_connections 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 1 -- 172.21.15.29:0/1309513608 wait complete. 2024-03-25T19:42:12.390 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.387+0000 7f7ef171c640 10 monclient: monmap: 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 librados: starting msgr at 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 librados: starting objecter 2024-03-25T19:42:12.391 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 Processor -- start 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 -- start start 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 librados: setting wanted keys 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 1 librados: calling monclient init 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.388+0000 7f7ef171c640 10 monclient: init 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-c con 0x7f7eec0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-a con 0x7f7eec13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient(hunting): picked mon.noname-b con 0x7f7eec1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec13c190 con 0x7f7eec13f710 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0b7cf0 con 0x7f7eec1430c0 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec13ad50 con 0x7f7eec0a98f0 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.389+0000 7f7ef171c640 10 monclient(hunting): authenticate will time out at 6609.143339s 2024-03-25T19:42:12.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.397+0000 7f7eea7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7eec13f710 0x7f7eec1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45352/0 (socket says 172.21.15.29:45352) 2024-03-25T19:42:12.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.397+0000 7f7eea7fc640 1 -- 172.21.15.29:0/2438612048 learned_addr learned my addr 172.21.15.29:0/2438612048 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:12.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.398+0000 7f7ed37fe640 10 client.?.objecter ms_handle_connect 0x7f7eec13f710 2024-03-25T19:42:12.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.398+0000 7f7ed37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.400+0000 7f7ed37fe640 10 client.?.objecter ms_handle_connect 0x7f7eec1430c0 2024-03-25T19:42:12.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.400+0000 7f7ed37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.400+0000 7f7ed37fe640 10 client.?.objecter ms_handle_connect 0x7f7eec0a98f0 2024-03-25T19:42:12.401 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.400+0000 7f7ed37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298679883 0 0) 0x7f7eec0b7cf0 con 0x7f7eec1430c0 2024-03-25T19:42:12.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:42:12.414 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.412+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed8004eb0 con 0x7f7eec1430c0 2024-03-25T19:42:12.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297870416 0 0) 0x7f7eec13ad50 con 0x7f7eec0a98f0 2024-03-25T19:42:12.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:42:12.416 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.415+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec0b7cf0 con 0x7f7eec0a98f0 2024-03-25T19:42:12.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865515040 0 0) 0x7f7ed4003520 con 0x7f7eec13f710 2024-03-25T19:42:12.422 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:42:12.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.422+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec13ad50 con 0x7f7eec13f710 2024-03-25T19:42:12.429 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954872432 0 0) 0x7f7ed8004eb0 con 0x7f7eec1430c0 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ed4003520 con 0x7f7eec1430c0 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f7ee4001ea0 con 0x7f7eec1430c0 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 10 monclient(hunting): dump: 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:12.430 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:12.431 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.431 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.429+0000 7f7ed37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:12.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2498820516 0 0) 0x7f7ed4003520 con 0x7f7eec1430c0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 >> v1:172.21.15.29:6790/0 conn(0x7f7eec0a98f0 legacy=0x7f7eec13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 >> v1:172.21.15.29:6789/0 conn(0x7f7eec13f710 legacy=0x7f7eec1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 1 monclient: found mon.b 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec13afc0 con 0x7f7eec1430c0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ef171c640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ef171c640 10 monclient: _renew_subs 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ef171c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:12.449 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.447+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec13b2a0 con 0x7f7eec1430c0 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.448+0000 7f7ef171c640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.448+0000 7f7ef171c640 10 monclient: _renew_subs 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.448+0000 7f7ef171c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.448+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec12fae0 con 0x7f7eec1430c0 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.448+0000 7f7ef171c640 1 librados: init done 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.449+0000 7f7ef171c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.450 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.449+0000 7f7ef171c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.451 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.449+0000 7f7ef171c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:12.451 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.449+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec0b3ef0 con 0x7f7eec1430c0 2024-03-25T19:42:12.739 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ee40020a0 con 0x7f7eec1430c0 2024-03-25T19:42:12.739 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f7ee4003dc0 con 0x7f7eec1430c0 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ed37fe640 10 monclient: dump: 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:12.740 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:12.741 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:12.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.738+0000 7f7ee97fa640 4 set_mon_vals no callback set 2024-03-25T19:42:12.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.739+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f7ee4019d90 con 0x7f7eec1430c0 2024-03-25T19:42:12.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.740+0000 7f7eeaffd640 10 monclient: get_auth_request con 0x7f7ed806dda0 auth_method 0 2024-03-25T19:42:12.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.740+0000 7f7ed37fe640 10 client.4603.objecter ms_handle_connect 0x7f7ed806dda0 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f7ee40539b0 con 0x7f7eec1430c0 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 10 client.4603.objecter ms_dispatch 0x7f7eec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ee404f2f0 con 0x7f7eec1430c0 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:12.744+0000 7f7ed37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:13.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.001+0000 7f7ef171c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.001+0000 7f7ef171c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.001+0000 7f7ef171c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:13.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.001+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7eec0627b0 con 0x7f7eec1430c0 2024-03-25T19:42:13.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.022+0000 7f7ed37fe640 1 -- 172.21.15.29:0/2438612048 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f7ee40513f0 con 0x7f7eec1430c0 2024-03-25T19:42:13.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.022+0000 7f7ed37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.023 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.022+0000 7f7ed37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:13.024 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:13.024 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 10 librados: watch_flush enter 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 10 librados: watch_flush exit 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 >> v1:172.21.15.29:6800/2314944989 conn(0x7f7ed806dda0 legacy=0x7f7ed8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 10 monclient: shutdown 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 >> v1:172.21.15.196:6789/0 conn(0x7f7eec1430c0 legacy=0x7f7eec1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 shutdown_connections 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 >> 172.21.15.29:0/2438612048 conn(0x7f7eec0b7ea0 msgr2=0x7f7eec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 shutdown_connections 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 -- 172.21.15.29:0/2438612048 wait complete. 2024-03-25T19:42:13.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:13.025+0000 7f7ef171c640 1 librados: shutdown 2024-03-25T19:42:13.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:19.843 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:19.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.956+0000 7f91d4c93640 10 monclient: get_monmap_and_config 2024-03-25T19:42:19.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.956+0000 7f91d4c93640 10 monclient: build_initial_monmap 2024-03-25T19:42:19.957 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.957+0000 7f91d4c93640 10 monclient: monmap: 2024-03-25T19:42:19.957 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:19.958809+0000 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:19.958809+0000 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:19.958 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:19.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.957+0000 7f91d4c93640 1 Processor -- start 2024-03-25T19:42:19.959 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.957+0000 7f91d4c93640 1 -- start start 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.957+0000 7f91d4c93640 10 monclient: init 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.958+0000 7f91d4c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.958+0000 7f91d4c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.958+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-a con 0x7f91d00a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.958+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-b con 0x7f91d013f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.958+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-c con 0x7f91d01430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.959+0000 7f91d4c93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d0142520 con 0x7f91d00a98f0 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.959+0000 7f91d4c93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b28a0 con 0x7f91d01430c0 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.959+0000 7f91d4c93640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b2a30 con 0x7f91d013f710 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.959+0000 7f91d4c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:19.960 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.959+0000 7f91d4c93640 10 monclient(hunting): authenticate will time out at 6616.713084s 2024-03-25T19:42:19.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.978+0000 7f91c7fff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f91d013f710 0x7f91d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:48290/0 (socket says 172.21.15.29:48290) 2024-03-25T19:42:19.979 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.978+0000 7f91c7fff640 1 -- 172.21.15.29:0/3202395118 learned_addr learned my addr 172.21.15.29:0/3202395118 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053715692 0 0) 0x7f91d00b2a30 con 0x7f91d013f710 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc003520 con 0x7f91d013f710 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449129918 0 0) 0x7f91d0142520 con 0x7f91d00a98f0 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:42:20.000 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:19.999+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d00b2a30 con 0x7f91d00a98f0 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776865978 0 0) 0x7f91d00b28a0 con 0x7f91d01430c0 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:42:20.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.004+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d0142520 con 0x7f91d01430c0 2024-03-25T19:42:20.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.026+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753211821 0 0) 0x7f91d0142520 con 0x7f91d01430c0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.026+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 >> v1:172.21.15.29:6790/0 conn(0x7f91d013f710 legacy=0x7f91d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.026+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 >> v1:172.21.15.29:6789/0 conn(0x7f91d00a98f0 legacy=0x7f91d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.026+0000 7f91cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.026+0000 7f91cdd9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00b44c0 con 0x7f91d01430c0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f91c8001930 con 0x7f91d01430c0 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:20.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91cdd9b640 10 monclient: dump: 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91d4c93640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:42:20.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.027+0000 7f91d4c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:20.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91c8001a90 con 0x7f91d01430c0 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 1 -- 172.21.15.29:0/3202395118 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f91c8001ee0 con 0x7f91d01430c0 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91d4c93640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91d4c93640 4 set_mon_vals no callback set 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91cdd9b640 10 monclient: dump: 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:20.042 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:20.043 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:20.043 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91d4c93640 10 monclient: shutdown 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91d4c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.041+0000 7f91d4c93640 1 -- 172.21.15.29:0/3202395118 >> v1:172.21.15.196:6789/0 conn(0x7f91d01430c0 legacy=0x7f91d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.042+0000 7f91d4c93640 1 -- 172.21.15.29:0/3202395118 shutdown_connections 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.042+0000 7f91d4c93640 1 -- 172.21.15.29:0/3202395118 >> 172.21.15.29:0/3202395118 conn(0x7f91d00b7ea0 msgr2=0x7f91d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.042+0000 7f91d4c93640 1 -- 172.21.15.29:0/3202395118 shutdown_connections 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.042+0000 7f91d4c93640 1 -- 172.21.15.29:0/3202395118 wait complete. 2024-03-25T19:42:20.044 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 10 monclient: build_initial_monmap 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 10 monclient: monmap: 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 librados: starting msgr at 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 librados: starting objecter 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 Processor -- start 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 -- start start 2024-03-25T19:42:20.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 librados: setting wanted keys 2024-03-25T19:42:20.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 1 librados: calling monclient init 2024-03-25T19:42:20.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.043+0000 7f91d4c93640 10 monclient: init 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-c con 0x7f91d00a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-a con 0x7f91d013f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient(hunting): picked mon.noname-b con 0x7f91d01430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d01d7800 con 0x7f91d013f710 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d01d83a0 con 0x7f91d01430c0 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00b04d0 con 0x7f91d00a98f0 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:20.047 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.044+0000 7f91d4c93640 10 monclient(hunting): authenticate will time out at 6616.798531s 2024-03-25T19:42:20.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.060+0000 7f91c7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91d013f710 0x7f91d0138c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33854/0 (socket says 172.21.15.29:33854) 2024-03-25T19:42:20.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.060+0000 7f91c7fff640 1 -- 172.21.15.29:0/3051533052 learned_addr learned my addr 172.21.15.29:0/3051533052 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:20.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.064+0000 7f91c6ffd640 10 client.?.objecter ms_handle_connect 0x7f91d013f710 2024-03-25T19:42:20.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.064+0000 7f91c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.068+0000 7f91c6ffd640 10 client.?.objecter ms_handle_connect 0x7f91d01430c0 2024-03-25T19:42:20.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.068+0000 7f91c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.073+0000 7f91c6ffd640 10 client.?.objecter ms_handle_connect 0x7f91d00a98f0 2024-03-25T19:42:20.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.073+0000 7f91c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378522440 0 0) 0x7f91d01d83a0 con 0x7f91d01430c0 2024-03-25T19:42:20.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:42:20.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.075+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0004eb0 con 0x7f91d01430c0 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774961472 0 0) 0x7f91d01d7800 con 0x7f91d013f710 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:42:20.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.083+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d01d83a0 con 0x7f91d013f710 2024-03-25T19:42:20.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220476682 0 0) 0x7f91b0004eb0 con 0x7f91d01430c0 2024-03-25T19:42:20.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d01d7800 con 0x7f91d01430c0 2024-03-25T19:42:20.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f91c8001de0 con 0x7f91d01430c0 2024-03-25T19:42:20.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.091+0000 7f91c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494822362 0 0) 0x7f91d00b04d0 con 0x7f91d00a98f0 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:42:20.097 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.096+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0004eb0 con 0x7f91d00a98f0 2024-03-25T19:42:20.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 69568194 0 0) 0x7f91d01d83a0 con 0x7f91d013f710 2024-03-25T19:42:20.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d00b04d0 con 0x7f91d013f710 2024-03-25T19:42:20.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f91b8001950 con 0x7f91d013f710 2024-03-25T19:42:20.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.100+0000 7f91c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:20.102 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3662992299 0 0) 0x7f91d01d7800 con 0x7f91d01430c0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 >> v1:172.21.15.29:6790/0 conn(0x7f91d00a98f0 legacy=0x7f91d01d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 >> v1:172.21.15.29:6789/0 conn(0x7f91d013f710 legacy=0x7f91d0138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 1 monclient: found mon.b 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0139a30 con 0x7f91d01430c0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91d4c93640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91d4c93640 10 monclient: _renew_subs 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91d4c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.115+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d00b0740 con 0x7f91d01430c0 2024-03-25T19:42:20.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.116+0000 7f91d4c93640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.116+0000 7f91d4c93640 10 monclient: _renew_subs 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.116+0000 7f91d4c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.116+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d01d7540 con 0x7f91d01430c0 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.116+0000 7f91d4c93640 1 librados: init done 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.117+0000 7f91d4c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.117+0000 7f91d4c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.117+0000 7f91d4c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.117+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d00627b0 con 0x7f91d01430c0 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91c8002020 con 0x7f91d01430c0 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f91c8003dc0 con 0x7f91d01430c0 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91c6ffd640 10 monclient: dump: 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:20.399 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.398+0000 7f91cd59a640 4 set_mon_vals no callback set 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.399+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f91c8019d90 con 0x7f91d01430c0 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.399+0000 7f91ce59c640 10 monclient: get_auth_request con 0x7f91b006dfb0 auth_method 0 2024-03-25T19:42:20.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.399+0000 7f91c6ffd640 10 client.4609.objecter ms_handle_connect 0x7f91b006dfb0 2024-03-25T19:42:20.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.404+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f91c80538e0 con 0x7f91d01430c0 2024-03-25T19:42:20.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.404+0000 7f91c6ffd640 10 client.4609.objecter ms_dispatch 0x7f91d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:20.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.404+0000 7f91c6ffd640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:20.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.404+0000 7f91c6ffd640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:20.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.404+0000 7f91c6ffd640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T19:42:20.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.405+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91c804f3f0 con 0x7f91d01430c0 2024-03-25T19:42:20.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.405+0000 7f91c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.405+0000 7f91c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:20.662 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.659+0000 7f91d4c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.662 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.659+0000 7f91d4c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.662 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.660+0000 7f91d4c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:20.662 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.660+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91d014e190 con 0x7f91d01430c0 2024-03-25T19:42:20.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.689+0000 7f91c6ffd640 1 -- 172.21.15.29:0/3051533052 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f91c8051320 con 0x7f91d01430c0 2024-03-25T19:42:20.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.689+0000 7f91c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.689+0000 7f91c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:20.690 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:20.690 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:20.692 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 10 librados: watch_flush enter 2024-03-25T19:42:20.692 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 10 librados: watch_flush exit 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 10 client.4609.objecter successfully canceled tick 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 >> v1:172.21.15.29:6800/2314944989 conn(0x7f91b006dfb0 legacy=0x7f91b0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 10 monclient: shutdown 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 >> v1:172.21.15.196:6789/0 conn(0x7f91d01430c0 legacy=0x7f91d0139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 shutdown_connections 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 >> 172.21.15.29:0/3051533052 conn(0x7f91d00b7ea0 msgr2=0x7f91d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.691+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 shutdown_connections 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.692+0000 7f91d4c93640 1 -- 172.21.15.29:0/3051533052 wait complete. 2024-03-25T19:42:20.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:20.692+0000 7f91d4c93640 1 librados: shutdown 2024-03-25T19:42:20.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:27.510 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:27.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.631+0000 7ff151d4b640 10 monclient: get_monmap_and_config 2024-03-25T19:42:27.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.631+0000 7ff151d4b640 10 monclient: build_initial_monmap 2024-03-25T19:42:27.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.631+0000 7ff151d4b640 10 monclient: monmap: 2024-03-25T19:42:27.631 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:27.631 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:27.632753+0000 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:27.632753+0000 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:27.632 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.632+0000 7ff151d4b640 1 Processor -- start 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.632+0000 7ff151d4b640 1 -- start start 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.632+0000 7ff151d4b640 10 monclient: init 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:27.633 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c142520 con 0x7ff14c13f710 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b28a0 con 0x7ff14c0a98f0 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b2a30 con 0x7ff14c1430c0 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:27.634 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.633+0000 7ff151d4b640 10 monclient(hunting): authenticate will time out at 6624.386655s 2024-03-25T19:42:27.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.640+0000 7ff14b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff14c0a98f0 0x7ff14c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:44692/0 (socket says 172.21.15.29:44692) 2024-03-25T19:42:27.640 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.640+0000 7ff14b7fe640 1 -- 172.21.15.29:0/485085526 learned_addr learned my addr 172.21.15.29:0/485085526 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:27.653 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707192983 0 0) 0x7ff14c0b2a30 con 0x7ff14c1430c0 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003520 con 0x7ff14c1430c0 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463515049 0 0) 0x7ff14c0b28a0 con 0x7ff14c0a98f0 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:42:27.654 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.654+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c0b2a30 con 0x7ff14c0a98f0 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945496566 0 0) 0x7ff14c142520 con 0x7ff14c13f710 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:42:27.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.657+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c0b28a0 con 0x7ff14c13f710 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266592851 0 0) 0x7ff14c0b2a30 con 0x7ff14c0a98f0 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 >> v1:172.21.15.29:6790/0 conn(0x7ff14c1430c0 legacy=0x7ff14c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 >> v1:172.21.15.29:6789/0 conn(0x7ff14c13f710 legacy=0x7ff14c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b44c0 con 0x7ff14c0a98f0 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:27.670 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff13c001670 con 0x7ff14c0a98f0 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff14a7fc640 10 monclient: dump: 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:27.671 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff151d4b640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:42:27.672 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.670+0000 7ff151d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff13c0017d0 con 0x7ff14c0a98f0 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 1 -- 172.21.15.29:0/485085526 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff13c001be0 con 0x7ff14c0a98f0 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff151d4b640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff151d4b640 4 set_mon_vals no callback set 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.688+0000 7ff14a7fc640 10 monclient: dump: 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.688 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.689 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:27.689 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:27.689 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:27.689 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:27.689 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 10 monclient: shutdown 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 1 -- 172.21.15.29:0/485085526 >> v1:172.21.15.196:6789/0 conn(0x7ff14c0a98f0 legacy=0x7ff14c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 1 -- 172.21.15.29:0/485085526 shutdown_connections 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 1 -- 172.21.15.29:0/485085526 >> 172.21.15.29:0/485085526 conn(0x7ff14c0b7ea0 msgr2=0x7ff14c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 1 -- 172.21.15.29:0/485085526 shutdown_connections 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.689+0000 7ff151d4b640 1 -- 172.21.15.29:0/485085526 wait complete. 2024-03-25T19:42:27.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.690+0000 7ff151d4b640 10 monclient: build_initial_monmap 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.690+0000 7ff151d4b640 10 monclient: monmap: 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.690+0000 7ff151d4b640 1 librados: starting msgr at 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.690+0000 7ff151d4b640 1 librados: starting objecter 2024-03-25T19:42:27.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 1 Processor -- start 2024-03-25T19:42:27.692 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 1 -- start start 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 1 librados: setting wanted keys 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 1 librados: calling monclient init 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 10 monclient: init 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.691+0000 7ff151d4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c139330 con 0x7ff14c0a98f0 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0abe70 con 0x7ff14c13f710 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0ac150 con 0x7ff14c1430c0 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:27.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.692+0000 7ff151d4b640 10 monclient(hunting): authenticate will time out at 6624.445039s 2024-03-25T19:42:27.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.708+0000 7ff14affd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff14c13f710 0x7ff14c0ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:44732/0 (socket says 172.21.15.29:44732) 2024-03-25T19:42:27.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.708+0000 7ff14affd640 1 -- 172.21.15.29:0/783679646 learned_addr learned my addr 172.21.15.29:0/783679646 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:27.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.710+0000 7ff133fff640 10 client.?.objecter ms_handle_connect 0x7ff14c13f710 2024-03-25T19:42:27.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.710+0000 7ff133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030065533 0 0) 0x7ff14c0abe70 con 0x7ff14c13f710 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff14c13f710 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 client.?.objecter ms_handle_connect 0x7ff14c1430c0 2024-03-25T19:42:27.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.716+0000 7ff133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.721+0000 7ff133fff640 10 client.?.objecter ms_handle_connect 0x7ff14c0a98f0 2024-03-25T19:42:27.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.721+0000 7ff133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2020179240 0 0) 0x7ff138004eb0 con 0x7ff14c13f710 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff14c0abe70 con 0x7ff14c13f710 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff140001a10 con 0x7ff14c13f710 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 10 monclient(hunting): dump: 2024-03-25T19:42:27.732 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.732+0000 7ff133fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:27.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925942666 0 0) 0x7ff14c0ac150 con 0x7ff14c1430c0 2024-03-25T19:42:27.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff14c1430c0 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496701520 0 0) 0x7ff14c139330 con 0x7ff14c0a98f0 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:42:27.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.737+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c0ac150 con 0x7ff14c0a98f0 2024-03-25T19:42:27.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512448721 0 0) 0x7ff14c0abe70 con 0x7ff14c13f710 2024-03-25T19:42:27.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 >> v1:172.21.15.29:6790/0 conn(0x7ff14c1430c0 legacy=0x7ff14c1e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 >> v1:172.21.15.29:6789/0 conn(0x7ff14c0a98f0 legacy=0x7ff14c1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 1 monclient: found mon.b 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0acc80 con 0x7ff14c13f710 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 10 monclient: _renew_subs 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c137e30 con 0x7ff14c13f710 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 10 monclient: _renew_subs 2024-03-25T19:42:27.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c136880 con 0x7ff14c13f710 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.744+0000 7ff151d4b640 1 librados: init done 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.745+0000 7ff151d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.745+0000 7ff151d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.745+0000 7ff151d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:27.746 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:27.745+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c0627b0 con 0x7ff14c13f710 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff140001cb0 con 0x7ff14c13f710 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7ff140003a60 con 0x7ff14c13f710 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff133fff640 10 monclient: dump: 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:28.035 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:28.036 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.035+0000 7ff149ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:28.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.036+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7ff140003c80 con 0x7ff14c13f710 2024-03-25T19:42:28.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.036+0000 7ff14b7fe640 10 monclient: get_auth_request con 0x7ff13806de80 auth_method 0 2024-03-25T19:42:28.037 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.036+0000 7ff133fff640 10 client.4615.objecter ms_handle_connect 0x7ff13806de80 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7ff1400548a0 con 0x7ff14c13f710 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 10 client.4615.objecter ms_dispatch 0x7ff14c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 3 client.4615.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 3 client.4615.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1400503b0 con 0x7ff14c13f710 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:28.041 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.041+0000 7ff133fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:28.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.304+0000 7ff151d4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.304+0000 7ff151d4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.304+0000 7ff151d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:28.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.304+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff14c1390e0 con 0x7ff14c13f710 2024-03-25T19:42:28.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.331+0000 7ff133fff640 1 -- 172.21.15.29:0/783679646 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7ff1400522e0 con 0x7ff14c13f710 2024-03-25T19:42:28.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.331+0000 7ff133fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.331 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.331+0000 7ff133fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:28.331 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:28.331 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:28.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 10 librados: watch_flush enter 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 10 librados: watch_flush exit 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 10 client.4615.objecter successfully canceled tick 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 >> v1:172.21.15.29:6800/2314944989 conn(0x7ff13806de80 legacy=0x7ff138070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 10 monclient: shutdown 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.333+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 >> v1:172.21.15.196:6789/0 conn(0x7ff14c13f710 legacy=0x7ff14c0ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.334+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 shutdown_connections 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.334+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 >> 172.21.15.29:0/783679646 conn(0x7ff14c0b7ea0 msgr2=0x7ff14c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.334+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 shutdown_connections 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.334+0000 7ff151d4b640 1 -- 172.21.15.29:0/783679646 wait complete. 2024-03-25T19:42:28.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:28.334+0000 7ff151d4b640 1 librados: shutdown 2024-03-25T19:42:28.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:31.144 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:01.307 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.153 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:01.317 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.197 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:01.361 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.218 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:01.380 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.227 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:01.389 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.231 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:01.395 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.240 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:01.404 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.247 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:01.410 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:31.268 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:01.431 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:35.153 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:35.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.270+0000 7fd171bc6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:35.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.270+0000 7fd171bc6640 10 monclient: build_initial_monmap 2024-03-25T19:42:35.270 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.270+0000 7fd171bc6640 10 monclient: monmap: 2024-03-25T19:42:35.270 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:35.270 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:35.271384+0000 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:35.271384+0000 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:35.271 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.271+0000 7fd171bc6640 1 Processor -- start 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.271+0000 7fd171bc6640 1 -- start start 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.271+0000 7fd171bc6640 10 monclient: init 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c0b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:35.272 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c14be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:35.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0af940 con 0x7fd16c14be90 2024-03-25T19:42:35.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0b0a70 con 0x7fd16c0b4d60 2024-03-25T19:42:35.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0b0d50 con 0x7fd16c0b1730 2024-03-25T19:42:35.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:35.273 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.272+0000 7fd171bc6640 10 monclient(hunting): authenticate will time out at 6632.025348s 2024-03-25T19:42:35.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.277+0000 7fd16bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd16c14be90 0x7fd16c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47326/0 (socket says 172.21.15.29:47326) 2024-03-25T19:42:35.276 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.277+0000 7fd16bfff640 1 -- 172.21.15.29:0/1958027988 learned_addr learned my addr 172.21.15.29:0/1958027988 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163527381 0 0) 0x7fd16c0af940 con 0x7fd16c14be90 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:42:35.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.288+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154003520 con 0x7fd16c14be90 2024-03-25T19:42:35.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311390676 0 0) 0x7fd154003520 con 0x7fd16c14be90 2024-03-25T19:42:35.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 >> v1:172.21.15.29:6790/0 conn(0x7fd16c0b1730 legacy=0x7fd16c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 >> v1:172.21.15.196:6789/0 conn(0x7fd16c0b4d60 legacy=0x7fd16c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0b1030 con 0x7fd16c14be90 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd164001a10 con 0x7fd16c14be90 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd171bc6640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd171bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd16a7fc640 10 monclient: dump: 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:35.299 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.300 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.298+0000 7fd171bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd164001b70 con 0x7fd16c14be90 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 1 -- 172.21.15.29:0/1958027988 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd164001f80 con 0x7fd16c14be90 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd16a7fc640 10 monclient: dump: 2024-03-25T19:42:35.309 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd171bc6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:35.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.309+0000 7fd171bc6640 4 set_mon_vals no callback set 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.310+0000 7fd171bc6640 10 monclient: shutdown 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.310+0000 7fd171bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.310+0000 7fd171bc6640 1 -- 172.21.15.29:0/1958027988 >> v1:172.21.15.29:6789/0 conn(0x7fd16c14be90 legacy=0x7fd16c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.311+0000 7fd171bc6640 1 -- 172.21.15.29:0/1958027988 shutdown_connections 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.311+0000 7fd171bc6640 1 -- 172.21.15.29:0/1958027988 >> 172.21.15.29:0/1958027988 conn(0x7fd16c0617a0 msgr2=0x7fd16c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:35.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.311+0000 7fd171bc6640 1 -- 172.21.15.29:0/1958027988 shutdown_connections 2024-03-25T19:42:35.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.311+0000 7fd171bc6640 1 -- 172.21.15.29:0/1958027988 wait complete. 2024-03-25T19:42:35.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 10 monclient: build_initial_monmap 2024-03-25T19:42:35.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 10 monclient: monmap: 2024-03-25T19:42:35.312 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:35.312 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 librados: starting msgr at 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 librados: starting objecter 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 Processor -- start 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 -- start start 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 librados: setting wanted keys 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 1 librados: calling monclient init 2024-03-25T19:42:35.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.312+0000 7fd171bc6640 10 monclient: init 2024-03-25T19:42:35.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:35.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:35.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:35.314 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c139dc0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c13a0a0 con 0x7fd16c0b1730 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c1425a0 con 0x7fd16c14be90 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:35.315 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.313+0000 7fd171bc6640 10 monclient(hunting): authenticate will time out at 6632.066657s 2024-03-25T19:42:35.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.327+0000 7fd16b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd16c0b1730 0x7fd16c142b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:49130/0 (socket says 172.21.15.29:49130) 2024-03-25T19:42:35.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.327+0000 7fd16affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd16c0b4d60 0x7fd16c13a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47356/0 (socket says 172.21.15.29:47356) 2024-03-25T19:42:35.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.327+0000 7fd16b7fe640 1 -- 172.21.15.29:0/2493568130 learned_addr learned my addr 172.21.15.29:0/2493568130 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:35.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.328+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c0b4d60 2024-03-25T19:42:35.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.328+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.328+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c0b1730 2024-03-25T19:42:35.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.328+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800752109 0 0) 0x7fd16c139dc0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:42:35.335 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.335+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c14be90 2024-03-25T19:42:35.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066588070 0 0) 0x7fd16c13a0a0 con 0x7fd16c0b1730 2024-03-25T19:42:35.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:42:35.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.338+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c139dc0 con 0x7fd16c0b1730 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1380339735 0 0) 0x7fd158004eb0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd16c13a0a0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd160001970 con 0x7fd16c0b4d60 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:35.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 10 monclient(hunting): dump: 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.345+0000 7fd153fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703245495 0 0) 0x7fd16c13a0a0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 >> v1:172.21.15.29:6790/0 conn(0x7fd16c14be90 legacy=0x7fd16c13abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 >> v1:172.21.15.196:6789/0 conn(0x7fd16c0b1730 legacy=0x7fd16c142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 1 monclient: found mon.a 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c138ed0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 10 monclient: _renew_subs 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c13b2e0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 10 monclient: _renew_subs 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c142080 con 0x7fd16c0b4d60 2024-03-25T19:42:35.355 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.354+0000 7fd171bc6640 1 librados: init done 2024-03-25T19:42:35.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.355+0000 7fd171bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.355+0000 7fd171bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.355+0000 7fd171bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.355+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c0b2270 con 0x7fd16c0b4d60 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd160003610 con 0x7fd16c0b4d60 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fd160003960 con 0x7fd16c0b4d60 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd153fff640 10 monclient: dump: 2024-03-25T19:42:35.677 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:35.678 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.677+0000 7fd169ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:35.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.678+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fd160003b80 con 0x7fd16c0b4d60 2024-03-25T19:42:35.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.678+0000 7fd16b7fe640 10 monclient: get_auth_request con 0x7fd15806dd10 auth_method 0 2024-03-25T19:42:35.679 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.679+0000 7fd153fff640 10 client.4620.objecter ms_handle_connect 0x7fd15806dd10 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fd160054750 con 0x7fd16c0b4d60 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 10 client.4620.objecter ms_dispatch 0x7fd16c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd160050260 con 0x7fd16c0b4d60 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.686+0000 7fd153fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:35.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.930+0000 7fd171bc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.930+0000 7fd171bc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.930+0000 7fd171bc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:35.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.930+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16c0627b0 con 0x7fd16c0b4d60 2024-03-25T19:42:35.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.951+0000 7fd153fff640 1 -- 172.21.15.29:0/2493568130 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fd160052190 con 0x7fd16c0b4d60 2024-03-25T19:42:35.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.951+0000 7fd153fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.951+0000 7fd153fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:35.951 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:35.951 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:35.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 10 librados: watch_flush enter 2024-03-25T19:42:35.953 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 10 librados: watch_flush exit 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 10 client.4620.objecter successfully canceled tick 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 >> v1:172.21.15.29:6800/2314944989 conn(0x7fd15806dd10 legacy=0x7fd1580701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 10 monclient: shutdown 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.953+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 >> v1:172.21.15.29:6789/0 conn(0x7fd16c0b4d60 legacy=0x7fd16c13a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.954+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 shutdown_connections 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.954+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 >> 172.21.15.29:0/2493568130 conn(0x7fd16c0617a0 msgr2=0x7fd16c13c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.954+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 shutdown_connections 2024-03-25T19:42:35.954 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.954+0000 7fd171bc6640 1 -- 172.21.15.29:0/2493568130 wait complete. 2024-03-25T19:42:35.955 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:35.954+0000 7fd171bc6640 1 librados: shutdown 2024-03-25T19:42:35.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:42.773 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.894+0000 7f4c13ca8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.894+0000 7f4c13ca8640 10 monclient: build_initial_monmap 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.894+0000 7f4c13ca8640 10 monclient: monmap: 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:42.895352+0000 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:42.895352+0000 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:42.894 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:42.895 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:42.895 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:42.895 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.895+0000 7f4c13ca8640 1 Processor -- start 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.895+0000 7f4c13ca8640 1 -- start start 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.895+0000 7f4c13ca8640 10 monclient: init 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.895+0000 7f4c13ca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f4c0c06adf0 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f4c0c143660 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c0c148c60 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:42.896 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0b1180 con 0x7f4c0c06adf0 2024-03-25T19:42:42.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0b13f0 con 0x7f4c0c143660 2024-03-25T19:42:42.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0afde0 con 0x7f4c0c148c60 2024-03-25T19:42:42.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:42.897 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.896+0000 7f4c13ca8640 10 monclient(hunting): authenticate will time out at 6639.649075s 2024-03-25T19:42:42.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.910+0000 7f4c11243640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4c0c143660 0x7f4c0c143af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:52788/0 (socket says 172.21.15.29:52788) 2024-03-25T19:42:42.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.910+0000 7f4c11243640 1 -- 172.21.15.29:0/2181458416 learned_addr learned my addr 172.21.15.29:0/2181458416 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424732708 0 0) 0x7f4c0c0b13f0 con 0x7f4c0c143660 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:42:42.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.925+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8003520 con 0x7f4c0c143660 2024-03-25T19:42:42.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833761114 0 0) 0x7f4c0c0afde0 con 0x7f4c0c148c60 2024-03-25T19:42:42.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:42:42.938 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.934+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c0b13f0 con 0x7f4c0c148c60 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676309357 0 0) 0x7f4c0c0b1180 con 0x7f4c0c06adf0 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:42:42.939 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.935+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c0afde0 con 0x7f4c0c06adf0 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089131359 0 0) 0x7f4bf8003520 con 0x7f4c0c143660 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 >> v1:172.21.15.29:6790/0 conn(0x7f4c0c148c60 legacy=0x7f4c0c14b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 >> v1:172.21.15.29:6789/0 conn(0x7f4c0c06adf0 legacy=0x7f4c0c0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: _finish_hunting 0 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 monclient: found mon.noname-c 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0b0300 con 0x7f4c0c143660 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: _finish_auth 0 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: _check_auth_tickets 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4bfc001690 con 0x7f4c0c143660 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c13ca8640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c13ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c10a42640 10 monclient: dump: 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.945+0000 7f4c13ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bfc0017f0 con 0x7f4c0c143660 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c10a42640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c10a42640 1 -- 172.21.15.29:0/2181458416 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4bfc001c00 con 0x7f4c0c143660 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c13ca8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c13ca8640 4 set_mon_vals no callback set 2024-03-25T19:42:42.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.964+0000 7f4c10a42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c10a42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c10a42640 10 monclient: dump: 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:42.965 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:42.966 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 10 monclient: shutdown 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 1 -- 172.21.15.29:0/2181458416 >> v1:172.21.15.196:6789/0 conn(0x7f4c0c143660 legacy=0x7f4c0c143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 1 -- 172.21.15.29:0/2181458416 shutdown_connections 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 1 -- 172.21.15.29:0/2181458416 >> 172.21.15.29:0/2181458416 conn(0x7f4c0c0b4b80 msgr2=0x7f4c0c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.965+0000 7f4c13ca8640 1 -- 172.21.15.29:0/2181458416 shutdown_connections 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.966+0000 7f4c13ca8640 1 -- 172.21.15.29:0/2181458416 wait complete. 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.966+0000 7f4c13ca8640 10 monclient: build_initial_monmap 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.966+0000 7f4c13ca8640 10 monclient: monmap: 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:42.967 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 librados: starting msgr at 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 librados: starting objecter 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 Processor -- start 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 -- start start 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 librados: setting wanted keys 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 librados: calling monclient init 2024-03-25T19:42:42.968 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient: init 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f4c0c06adf0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f4c0c143660 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f4c0c148c60 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c13f3f0 con 0x7f4c0c143660 2024-03-25T19:42:42.969 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c1316c0 con 0x7f4c0c148c60 2024-03-25T19:42:42.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c13eb60 con 0x7f4c0c06adf0 2024-03-25T19:42:42.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:42.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.967+0000 7f4c13ca8640 10 monclient(hunting): authenticate will time out at 6639.720821s 2024-03-25T19:42:42.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.981+0000 7f4c11243640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4c0c143660 0x7f4c0c1308a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57502/0 (socket says 172.21.15.29:57502) 2024-03-25T19:42:42.980 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.981+0000 7f4c11243640 1 -- 172.21.15.29:0/4171845609 learned_addr learned my addr 172.21.15.29:0/4171845609 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:42.981 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.982+0000 7f4c027fc640 10 client.?.objecter ms_handle_connect 0x7f4c0c143660 2024-03-25T19:42:42.982 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.982+0000 7f4c027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304774835 0 0) 0x7f4c0c13f3f0 con 0x7f4c0c143660 2024-03-25T19:42:42.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.986 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:42:42.987 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.987+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf0004eb0 con 0x7f4c0c143660 2024-03-25T19:42:42.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.989+0000 7f4c027fc640 10 client.?.objecter ms_handle_connect 0x7f4c0c06adf0 2024-03-25T19:42:42.989 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.989+0000 7f4c027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.991 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.992+0000 7f4c027fc640 10 client.?.objecter ms_handle_connect 0x7f4c0c148c60 2024-03-25T19:42:42.991 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.992+0000 7f4c027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668884028 0 0) 0x7f4c0c13eb60 con 0x7f4c0c06adf0 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:42:42.998 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:42.998+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c13f3f0 con 0x7f4c0c06adf0 2024-03-25T19:42:43.002 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2380769330 0 0) 0x7f4bf0004eb0 con 0x7f4c0c143660 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c0c13eb60 con 0x7f4c0c143660 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035305809 0 0) 0x7f4c0c1316c0 con 0x7f4c0c148c60 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:42:43.003 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.003+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf0004eb0 con 0x7f4c0c148c60 2024-03-25T19:42:43.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.005+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4bfc001c60 con 0x7f4c0c143660 2024-03-25T19:42:43.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.005+0000 7f4c027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.005+0000 7f4c027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:43.004 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.005+0000 7f4c027fc640 10 monclient(hunting): dump: 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:43.005 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.005+0000 7f4c027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189298185 0 0) 0x7f4c0c13f3f0 con 0x7f4c0c06adf0 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c0c1316c0 con 0x7f4c0c06adf0 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4c08001870 con 0x7f4c0c06adf0 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.012+0000 7f4c027fc640 10 monclient(hunting): dump: 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:43.012 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:43.013 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2432132777 0 0) 0x7f4c0c13eb60 con 0x7f4c0c143660 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 >> v1:172.21.15.29:6790/0 conn(0x7f4c0c06adf0 legacy=0x7f4c0c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 >> v1:172.21.15.196:6789/0 conn(0x7f4c0c148c60 legacy=0x7f4c0c130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 1 monclient: found mon.a 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c13dee0 con 0x7f4c0c143660 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c13ca8640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.015+0000 7f4c13ca8640 10 monclient: _renew_subs 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:43.016 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c13ff90 con 0x7f4c0c143660 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 10 monclient: _renew_subs 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c1338e0 con 0x7f4c0c143660 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.016+0000 7f4c13ca8640 1 librados: init done 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.017+0000 7f4c13ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.017+0000 7f4c13ca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.017+0000 7f4c13ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:43.017 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.017+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c13f5a0 con 0x7f4c0c143660 2024-03-25T19:42:43.328 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bfc0011f0 con 0x7f4c0c143660 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4bfc004b10 con 0x7f4c0c143660 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 10 monclient: dump: 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:43.329 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c03fff640 4 set_mon_vals no callback set 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.329+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f4bfc01ab60 con 0x7f4c0c143660 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.330+0000 7f4c11a44640 10 monclient: get_auth_request con 0x7f4bf006dee0 auth_method 0 2024-03-25T19:42:43.330 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.330+0000 7f4c027fc640 10 client.4626.objecter ms_handle_connect 0x7f4bf006dee0 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f4bfc054880 con 0x7f4c0c143660 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 10 client.4626.objecter ms_dispatch 0x7f4c0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bfc0501c0 con 0x7f4c0c143660 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.336+0000 7f4c027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:43.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.565+0000 7f4c13ca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.565+0000 7f4c13ca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.565+0000 7f4c13ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:42:43.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.565+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c0c06b1d0 con 0x7f4c0c143660 2024-03-25T19:42:43.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.589+0000 7f4c027fc640 1 -- 172.21.15.29:0/4171845609 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f4bfc0522c0 con 0x7f4c0c143660 2024-03-25T19:42:43.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.589+0000 7f4c027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.589 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.589+0000 7f4c027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:43.589 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:43.589 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 10 librados: watch_flush enter 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 10 librados: watch_flush exit 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 10 client.4626.objecter successfully canceled tick 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 >> v1:172.21.15.29:6800/2314944989 conn(0x7f4bf006dee0 legacy=0x7f4bf00703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 10 monclient: shutdown 2024-03-25T19:42:43.591 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.591+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 >> v1:172.21.15.29:6789/0 conn(0x7f4c0c143660 legacy=0x7f4c0c1308a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.592+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 shutdown_connections 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.592+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 >> 172.21.15.29:0/4171845609 conn(0x7f4c0c0b4b80 msgr2=0x7f4c0c071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.592+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 shutdown_connections 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.592+0000 7f4c13ca8640 1 -- 172.21.15.29:0/4171845609 wait complete. 2024-03-25T19:42:43.592 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:43.592+0000 7f4c13ca8640 1 librados: shutdown 2024-03-25T19:42:43.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:50.410 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.526+0000 7f80f5caf640 10 monclient: get_monmap_and_config 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.526+0000 7f80f5caf640 10 monclient: build_initial_monmap 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.526+0000 7f80f5caf640 10 monclient: monmap: 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:50.527092+0000 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:50.527092+0000 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:50.526 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:50.527 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:50.527 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.527+0000 7f80f5caf640 1 Processor -- start 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.527+0000 7f80f5caf640 1 -- start start 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.527+0000 7f80f5caf640 10 monclient: init 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-a con 0x7f80f00a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-c con 0x7f80f013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-b con 0x7f80f01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f0142520 con 0x7f80f00a98f0 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f00b28a0 con 0x7f80f013f710 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f00b2a30 con 0x7f80f01430c0 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:50.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.528+0000 7f80f5caf640 10 monclient(hunting): authenticate will time out at 6647.281311s 2024-03-25T19:42:50.536 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.537+0000 7f80effff640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f80f01430c0 0x7f80f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:40958/0 (socket says 172.21.15.29:40958) 2024-03-25T19:42:50.536 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.537+0000 7f80effff640 1 -- 172.21.15.29:0/4154822545 learned_addr learned my addr 172.21.15.29:0/4154822545 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:50.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.550+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160487768 0 0) 0x7f80f00b28a0 con 0x7f80f013f710 2024-03-25T19:42:50.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.551+0000 7f80ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.551+0000 7f80ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.551+0000 7f80ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.551+0000 7f80ee7fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:42:50.551 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.551+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d8003520 con 0x7f80f013f710 2024-03-25T19:42:50.559 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726325755 0 0) 0x7f80f00b2a30 con 0x7f80f01430c0 2024-03-25T19:42:50.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:42:50.560 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.560+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f00b28a0 con 0x7f80f01430c0 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513366467 0 0) 0x7f80d8003520 con 0x7f80f013f710 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 >> v1:172.21.15.29:6790/0 conn(0x7f80f01430c0 legacy=0x7f80f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 >> v1:172.21.15.29:6789/0 conn(0x7f80f00a98f0 legacy=0x7f80f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00b44c0 con 0x7f80f013f710 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:50.562 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f80e4001690 con 0x7f80f013f710 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80ee7fc640 10 monclient: dump: 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80f5caf640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:42:50.563 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.562+0000 7f80f5caf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:50.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.565+0000 7f80ee7fc640 1 -- 172.21.15.29:0/4154822545 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80e40017f0 con 0x7f80f013f710 2024-03-25T19:42:50.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.565+0000 7f80ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:50.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.565+0000 7f80f5caf640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:50.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.565+0000 7f80f5caf640 4 set_mon_vals no callback set 2024-03-25T19:42:50.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 10 monclient: shutdown 2024-03-25T19:42:50.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:50.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 1 -- 172.21.15.29:0/4154822545 >> v1:172.21.15.196:6789/0 conn(0x7f80f013f710 legacy=0x7f80f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 1 -- 172.21.15.29:0/4154822545 shutdown_connections 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 1 -- 172.21.15.29:0/4154822545 >> 172.21.15.29:0/4154822545 conn(0x7f80f00b7ea0 msgr2=0x7f80f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 1 -- 172.21.15.29:0/4154822545 shutdown_connections 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.566+0000 7f80f5caf640 1 -- 172.21.15.29:0/4154822545 wait complete. 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.567+0000 7f80f5caf640 10 monclient: build_initial_monmap 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.567+0000 7f80f5caf640 10 monclient: monmap: 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:50.567 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:50.568 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:50.568 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.567+0000 7f80f5caf640 1 librados: starting msgr at 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.567+0000 7f80f5caf640 1 librados: starting objecter 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.567+0000 7f80f5caf640 1 Processor -- start 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 -- start start 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 librados: setting wanted keys 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 librados: calling monclient init 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient: init 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:50.569 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-a con 0x7f80f00a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-b con 0x7f80f013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient(hunting): picked mon.noname-c con 0x7f80f01430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f0139a00 con 0x7f80f00a98f0 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f0135300 con 0x7f80f013f710 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f0135490 con 0x7f80f01430c0 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:50.570 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.568+0000 7f80f5caf640 10 monclient(hunting): authenticate will time out at 6647.321483s 2024-03-25T19:42:50.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.578+0000 7f80eeffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f80f013f710 0x7f80f01d4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36226/0 (socket says 172.21.15.29:36226) 2024-03-25T19:42:50.578 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.578+0000 7f80eeffd640 1 -- 172.21.15.29:0/3245056041 learned_addr learned my addr 172.21.15.29:0/3245056041 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:50.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.579+0000 7f80d7fff640 10 client.?.objecter ms_handle_connect 0x7f80f013f710 2024-03-25T19:42:50.579 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.579+0000 7f80d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684286413 0 0) 0x7f80f0135300 con 0x7f80f013f710 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:42:50.586 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.587+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80dc004eb0 con 0x7f80f013f710 2024-03-25T19:42:50.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.589+0000 7f80d7fff640 10 client.?.objecter ms_handle_connect 0x7f80f01430c0 2024-03-25T19:42:50.588 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.589+0000 7f80d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.594+0000 7f80d7fff640 10 client.?.objecter ms_handle_connect 0x7f80f00a98f0 2024-03-25T19:42:50.593 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.594+0000 7f80d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194351626 0 0) 0x7f80f0139a00 con 0x7f80f00a98f0 2024-03-25T19:42:50.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.602 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:42:50.603 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.602+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0135300 con 0x7f80f00a98f0 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123321952 0 0) 0x7f80f00b28a0 con 0x7f80f01430c0 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:42:50.605 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.605+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0139a00 con 0x7f80f01430c0 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924725923 0 0) 0x7f80dc004eb0 con 0x7f80f013f710 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80f00b28a0 con 0x7f80f013f710 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f80e4001d60 con 0x7f80f013f710 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:50.614 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.615 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.614+0000 7f80d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 317440397 0 0) 0x7f80f0135300 con 0x7f80f00a98f0 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80dc004eb0 con 0x7f80f00a98f0 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f80e00018b0 con 0x7f80f00a98f0 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.619+0000 7f80d7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.619 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.620 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:50.620 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:50.620 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:50.620 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:50.620 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939844085 0 0) 0x7f80f00b28a0 con 0x7f80f013f710 2024-03-25T19:42:50.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 >> v1:172.21.15.29:6790/0 conn(0x7f80f01430c0 legacy=0x7f80f01e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.629 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 >> v1:172.21.15.29:6789/0 conn(0x7f80f00a98f0 legacy=0x7f80f0138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 1 monclient: found mon.b 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0139080 con 0x7f80f013f710 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 10 monclient: _renew_subs 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f01d4f40 con 0x7f80f013f710 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:50.630 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 10 monclient: _renew_subs 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f0138400 con 0x7f80f013f710 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.629+0000 7f80f5caf640 1 librados: init done 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.630+0000 7f80f5caf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.630+0000 7f80f5caf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.630+0000 7f80f5caf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:50.631 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.630+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f00627b0 con 0x7f80f013f710 2024-03-25T19:42:50.928 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80e40038a0 con 0x7f80f013f710 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f80e4001250 con 0x7f80f013f710 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 10 monclient: dump: 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:50.929 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80edffb640 4 set_mon_vals no callback set 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.929+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f80e4003d70 con 0x7f80f013f710 2024-03-25T19:42:50.930 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.930+0000 7f80ef7fe640 10 monclient: get_auth_request con 0x7f80dc06df10 auth_method 0 2024-03-25T19:42:50.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.930+0000 7f80d7fff640 10 client.4633.objecter ms_handle_connect 0x7f80dc06df10 2024-03-25T19:42:50.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.932+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f80e40548f0 con 0x7f80f013f710 2024-03-25T19:42:50.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.932+0000 7f80d7fff640 10 client.4633.objecter ms_dispatch 0x7f80f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:50.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.932+0000 7f80d7fff640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:50.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.932+0000 7f80d7fff640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:50.932 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.932+0000 7f80d7fff640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T19:42:50.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.935+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80e4050400 con 0x7f80f013f710 2024-03-25T19:42:50.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.935+0000 7f80d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:50.935+0000 7f80d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:51.169 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.170+0000 7f80f5caf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.170+0000 7f80f5caf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.170+0000 7f80f5caf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:42:51.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.170+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80f0139210 con 0x7f80f013f710 2024-03-25T19:42:51.223 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:21.387 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.230 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:21.394 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.263 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:21.426 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.280 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:21.442 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.295 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:21.459 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.300 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:21.464 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.300 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:21.463 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.309+0000 7f80d7fff640 1 -- 172.21.15.29:0/3245056041 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f80e4052330 con 0x7f80f013f710 2024-03-25T19:42:51.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.309+0000 7f80d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.309+0000 7f80d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:51.310 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:51.310 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 10 librados: watch_flush enter 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 10 librados: watch_flush exit 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 10 client.4633.objecter successfully canceled tick 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 >> v1:172.21.15.29:6800/2314944989 conn(0x7f80dc06df10 legacy=0x7f80dc0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 10 monclient: shutdown 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:51.312 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 >> v1:172.21.15.196:6789/0 conn(0x7f80f013f710 legacy=0x7f80f01d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 shutdown_connections 2024-03-25T19:42:51.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 >> 172.21.15.29:0/3245056041 conn(0x7f80f00b7ea0 msgr2=0x7f80f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:51.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 shutdown_connections 2024-03-25T19:42:51.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 -- 172.21.15.29:0/3245056041 wait complete. 2024-03-25T19:42:51.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:51.312+0000 7f80f5caf640 1 librados: shutdown 2024-03-25T19:42:51.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:51.326 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:21.489 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:51.333 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:21.496 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:58.129 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:58.244 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.245+0000 7fe166b97640 10 monclient: get_monmap_and_config 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.245+0000 7fe166b97640 10 monclient: build_initial_monmap 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.245+0000 7fe166b97640 10 monclient: monmap: 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:42:58.245963+0000 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:42:58.245963+0000 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:42:58.245 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:42:58.246 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 1 Processor -- start 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 1 -- start start 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 10 monclient: init 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.246+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-c con 0x7fe1600b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:58.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-b con 0x7fe1600b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-a con 0x7fe16014be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600af940 con 0x7fe16014be90 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b0a70 con 0x7fe1600b4d60 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b0d50 con 0x7fe1600b1730 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:58.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.247+0000 7fe166b97640 10 monclient(hunting): authenticate will time out at 6654.999950s 2024-03-25T19:42:58.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.258+0000 7fe164933640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe1600b4d60 0x7fe16013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:58554/0 (socket says 172.21.15.29:58554) 2024-03-25T19:42:58.258 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.258+0000 7fe164933640 1 -- 172.21.15.29:0/1347178539 learned_addr learned my addr 172.21.15.29:0/1347178539 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:58.278 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796552910 0 0) 0x7fe1600b0a70 con 0x7fe1600b4d60 2024-03-25T19:42:58.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:42:58.279 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.279+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe150003520 con 0x7fe1600b4d60 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157803103 0 0) 0x7fe1600af940 con 0x7fe16014be90 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:42:58.284 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.284+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1600b0a70 con 0x7fe16014be90 2024-03-25T19:42:58.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.287+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680877107 0 0) 0x7fe1600b0d50 con 0x7fe1600b1730 2024-03-25T19:42:58.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.288+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.288+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.288+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.287 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.288+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:42:58.288 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.288+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1600af940 con 0x7fe1600b1730 2024-03-25T19:42:58.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085835356 0 0) 0x7fe150003520 con 0x7fe1600b4d60 2024-03-25T19:42:58.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 >> v1:172.21.15.29:6790/0 conn(0x7fe1600b1730 legacy=0x7fe1600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 >> v1:172.21.15.29:6789/0 conn(0x7fe16014be90 legacy=0x7fe16014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.296 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600b1030 con 0x7fe1600b4d60 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe154001690 con 0x7fe1600b4d60 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe15f7fe640 10 monclient: dump: 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:58.297 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe166b97640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:42:58.298 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.296+0000 7fe166b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:58.302 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1540017f0 con 0x7fe1600b4d60 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 1 -- 172.21.15.29:0/1347178539 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe154001c00 con 0x7fe1600b4d60 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe166b97640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe166b97640 4 set_mon_vals no callback set 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.303+0000 7fe15f7fe640 10 monclient: dump: 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.303 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.304 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:58.304 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:58.304 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:58.304 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:58.304 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 10 monclient: shutdown 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 1 -- 172.21.15.29:0/1347178539 >> v1:172.21.15.196:6789/0 conn(0x7fe1600b4d60 legacy=0x7fe16013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 1 -- 172.21.15.29:0/1347178539 shutdown_connections 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 1 -- 172.21.15.29:0/1347178539 >> 172.21.15.29:0/1347178539 conn(0x7fe1600617a0 msgr2=0x7fe160061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 1 -- 172.21.15.29:0/1347178539 shutdown_connections 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.304+0000 7fe166b97640 1 -- 172.21.15.29:0/1347178539 wait complete. 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 10 monclient: build_initial_monmap 2024-03-25T19:42:58.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 10 monclient: monmap: 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 librados: starting msgr at 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 librados: starting objecter 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 Processor -- start 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 -- start start 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 librados: setting wanted keys 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 1 librados: calling monclient init 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.305+0000 7fe166b97640 10 monclient: init 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-b con 0x7fe1600b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-a con 0x7fe1600b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient(hunting): picked mon.noname-c con 0x7fe16014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160140d00 con 0x7fe1600b4d60 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160141cc0 con 0x7fe1600b1730 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1601d8470 con 0x7fe16014be90 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.306+0000 7fe166b97640 10 monclient(hunting): authenticate will time out at 6655.059571s 2024-03-25T19:42:58.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.308+0000 7fe165134640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fe16014be90 0x7fe160134d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:49512/0 (socket says 172.21.15.29:49512) 2024-03-25T19:42:58.309 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.308+0000 7fe165134640 1 -- 172.21.15.29:0/938138179 learned_addr learned my addr 172.21.15.29:0/938138179 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:42:58.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.310+0000 7fe15d7fa640 10 client.?.objecter ms_handle_connect 0x7fe16014be90 2024-03-25T19:42:58.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.310+0000 7fe15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.313+0000 7fe15d7fa640 10 client.?.objecter ms_handle_connect 0x7fe1600b1730 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.313+0000 7fe15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.317+0000 7fe15d7fa640 10 client.?.objecter ms_handle_connect 0x7fe1600b4d60 2024-03-25T19:42:58.317 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.317+0000 7fe15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494579296 0 0) 0x7fe1601d8470 con 0x7fe16014be90 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:42:58.318 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.318+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe144004eb0 con 0x7fe16014be90 2024-03-25T19:42:58.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.325+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212236808 0 0) 0x7fe160141cc0 con 0x7fe1600b1730 2024-03-25T19:42:58.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.325 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1601d8470 con 0x7fe1600b1730 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089854456 0 0) 0x7fe144004eb0 con 0x7fe16014be90 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160141cc0 con 0x7fe16014be90 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe158001c90 con 0x7fe16014be90 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.326 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): dump: 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.327 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.326+0000 7fe15d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62802433 0 0) 0x7fe160140d00 con 0x7fe1600b4d60 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:42:58.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.334+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe144004eb0 con 0x7fe1600b4d60 2024-03-25T19:42:58.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501762706 0 0) 0x7fe160141cc0 con 0x7fe16014be90 2024-03-25T19:42:58.339 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 >> v1:172.21.15.196:6789/0 conn(0x7fe1600b1730 legacy=0x7fe1601405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 >> v1:172.21.15.29:6789/0 conn(0x7fe1600b4d60 legacy=0x7fe1601d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 1 monclient: found mon.c 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1601d8600 con 0x7fe16014be90 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe15d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe166b97640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe166b97640 10 monclient: _renew_subs 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe166b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.339+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16013b3c0 con 0x7fe16014be90 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:58.340 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 monclient: _renew_subs 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1601425a0 con 0x7fe16014be90 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 1 librados: init done 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:42:58.341 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.340+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1600b2270 con 0x7fe16014be90 2024-03-25T19:42:58.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe158003970 con 0x7fe16014be90 2024-03-25T19:42:58.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:58.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe158003d80 con 0x7fe16014be90 2024-03-25T19:42:58.650 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15d7fa640 10 monclient: dump: 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:42:58.651 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.650+0000 7fe15effd640 4 set_mon_vals no callback set 2024-03-25T19:42:58.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.651+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fe158003fa0 con 0x7fe16014be90 2024-03-25T19:42:58.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.651+0000 7fe164933640 10 monclient: get_auth_request con 0x7fe14406de00 auth_method 0 2024-03-25T19:42:58.652 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.652+0000 7fe15d7fa640 10 client.4628.objecter ms_handle_connect 0x7fe14406de00 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fe158054a10 con 0x7fe16014be90 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 10 client.4628.objecter ms_dispatch 0x7fe160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe158050520 con 0x7fe16014be90 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.657 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.657+0000 7fe15d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:58.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.916+0000 7fe166b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.916+0000 7fe166b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.916+0000 7fe166b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:42:58.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.916+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1600627b0 con 0x7fe16014be90 2024-03-25T19:42:58.941 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.942+0000 7fe15d7fa640 1 -- 172.21.15.29:0/938138179 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fe158052450 con 0x7fe16014be90 2024-03-25T19:42:58.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.942+0000 7fe15d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.942 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.942+0000 7fe15d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:58.942 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:42:58.942 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 10 librados: watch_flush enter 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 10 librados: watch_flush exit 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 >> v1:172.21.15.29:6800/2314944989 conn(0x7fe14406de00 legacy=0x7fe1440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 10 monclient: shutdown 2024-03-25T19:42:58.944 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 >> v1:172.21.15.29:6790/0 conn(0x7fe16014be90 legacy=0x7fe160134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 shutdown_connections 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 >> 172.21.15.29:0/938138179 conn(0x7fe1600617a0 msgr2=0x7fe1600b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 shutdown_connections 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 -- 172.21.15.29:0/938138179 wait complete. 2024-03-25T19:42:58.945 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:42:58.944+0000 7fe166b97640 1 librados: shutdown 2024-03-25T19:42:58.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:05.761 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:05.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.875+0000 7fbeef3d0640 10 monclient: get_monmap_and_config 2024-03-25T19:43:05.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.875+0000 7fbeef3d0640 10 monclient: build_initial_monmap 2024-03-25T19:43:05.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.875+0000 7fbeef3d0640 10 monclient: monmap: 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:05.876724+0000 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:05.876724+0000 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:05.876 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.876+0000 7fbeef3d0640 1 Processor -- start 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.876+0000 7fbeef3d0640 1 -- start start 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.876+0000 7fbeef3d0640 10 monclient: init 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-b con 0x7fbee80a96e0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbee80b1b10 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbee8146060 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee8140040 con 0x7fbee80b1b10 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80b1fa0 con 0x7fbee8146060 2024-03-25T19:43:05.878 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80b24c0 con 0x7fbee80a96e0 2024-03-25T19:43:05.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:05.879 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.877+0000 7fbeef3d0640 10 monclient(hunting): authenticate will time out at 6662.630367s 2024-03-25T19:43:05.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.893+0000 7fbeed96d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbee8146060 0x7fbee814dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:34878/0 (socket says 172.21.15.29:34878) 2024-03-25T19:43:05.893 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.893+0000 7fbeed96d640 1 -- 172.21.15.29:0/873418563 learned_addr learned my addr 172.21.15.29:0/873418563 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:05.900 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759239265 0 0) 0x7fbee80b1fa0 con 0x7fbee8146060 2024-03-25T19:43:05.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:43:05.901 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.901+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4003520 con 0x7fbee8146060 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575373445 0 0) 0x7fbed4003520 con 0x7fbee8146060 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 >> v1:172.21.15.29:6790/0 conn(0x7fbee80a96e0 legacy=0x7fbee80b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 >> v1:172.21.15.29:6789/0 conn(0x7fbee80b1b10 legacy=0x7fbee8145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 monclient: found mon.noname-c 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80b15b0 con 0x7fbee8146060 2024-03-25T19:43:05.913 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: _finish_auth 0 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbee0001a10 con 0x7fbee8146060 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbeef3d0640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbeef3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbedffff640 10 monclient: dump: 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.914 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:05.915 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:05.915 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:05.915 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:05.915 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.913+0000 7fbeef3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001b70 con 0x7fbee8146060 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 1 -- 172.21.15.29:0/873418563 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbee0001f80 con 0x7fbee8146060 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbeef3d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbeef3d0640 4 set_mon_vals no callback set 2024-03-25T19:43:05.919 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.919+0000 7fbedffff640 10 monclient: dump: 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:05.920 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 10 monclient: shutdown 2024-03-25T19:43:05.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:05.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 1 -- 172.21.15.29:0/873418563 >> v1:172.21.15.196:6789/0 conn(0x7fbee8146060 legacy=0x7fbee814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 1 -- 172.21.15.29:0/873418563 shutdown_connections 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 1 -- 172.21.15.29:0/873418563 >> 172.21.15.29:0/873418563 conn(0x7fbee80b7ea0 msgr2=0x7fbee80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.920+0000 7fbeef3d0640 1 -- 172.21.15.29:0/873418563 shutdown_connections 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.921+0000 7fbeef3d0640 1 -- 172.21.15.29:0/873418563 wait complete. 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.921+0000 7fbeef3d0640 10 monclient: build_initial_monmap 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.921+0000 7fbeef3d0640 10 monclient: monmap: 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.921+0000 7fbeef3d0640 1 librados: starting msgr at 2024-03-25T19:43:05.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.921+0000 7fbeef3d0640 1 librados: starting objecter 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 Processor -- start 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 -- start start 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 librados: setting wanted keys 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 librados: calling monclient init 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient: init 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbee80a96e0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbee8146060 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient(hunting): picked mon.noname-b con 0x7fbee813e7a0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80ad6c0 con 0x7fbee80a96e0 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80b08e0 con 0x7fbee813e7a0 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee81205b0 con 0x7fbee8146060 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:05.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.922+0000 7fbeef3d0640 10 monclient(hunting): authenticate will time out at 6662.675829s 2024-03-25T19:43:05.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.938+0000 7fbeec96b640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fbee8146060 0x7fbee813e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:53966/0 (socket says 172.21.15.29:53966) 2024-03-25T19:43:05.937 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.938+0000 7fbeec96b640 1 -- 172.21.15.29:0/896575879 learned_addr learned my addr 172.21.15.29:0/896575879 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:05.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.940+0000 7fbeddffb640 10 client.?.objecter ms_handle_connect 0x7fbee8146060 2024-03-25T19:43:05.940 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.940+0000 7fbeddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.943+0000 7fbeddffb640 10 client.?.objecter ms_handle_connect 0x7fbee813e7a0 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.943+0000 7fbeddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542747992 0 0) 0x7fbee81205b0 con 0x7fbee8146060 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.947+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc004eb0 con 0x7fbee8146060 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907514318 0 0) 0x7fbee80b08e0 con 0x7fbee813e7a0 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:43:05.949 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.949+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee81205b0 con 0x7fbee813e7a0 2024-03-25T19:43:05.951 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.952+0000 7fbeddffb640 10 client.?.objecter ms_handle_connect 0x7fbee80a96e0 2024-03-25T19:43:05.952 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.952+0000 7fbeddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053005654 0 0) 0x7fbee81205b0 con 0x7fbee813e7a0 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee80b08e0 con 0x7fbee813e7a0 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbee0001e80 con 0x7fbee813e7a0 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:05.963 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 10 monclient(hunting): dump: 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.964 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.963+0000 7fbeddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.970+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732030660 0 0) 0x7fbecc004eb0 con 0x7fbee8146060 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee81205b0 con 0x7fbee8146060 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbed8001a40 con 0x7fbee8146060 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient(hunting): dump: 2024-03-25T19:43:05.971 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2841783262 0 0) 0x7fbee80b08e0 con 0x7fbee813e7a0 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 >> v1:172.21.15.29:6790/0 conn(0x7fbee8146060 legacy=0x7fbee813e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 >> v1:172.21.15.29:6789/0 conn(0x7fbee80a96e0 legacy=0x7fbee80b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 monclient: found mon.b 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:05.972 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee81215f0 con 0x7fbee813e7a0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 10 monclient: _renew_subs 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee811ff40 con 0x7fbee813e7a0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 10 monclient: _renew_subs 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee81d8160 con 0x7fbee813e7a0 2024-03-25T19:43:05.973 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.971+0000 7fbeef3d0640 1 librados: init done 2024-03-25T19:43:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.972+0000 7fbeef3d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.972+0000 7fbeef3d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.972+0000 7fbeef3d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:05.974 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:05.972+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee8120420 con 0x7fbee813e7a0 2024-03-25T19:43:06.247 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee0002080 con 0x7fbee813e7a0 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbee0004fb0 con 0x7fbee813e7a0 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbeddffb640 10 monclient: dump: 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:06.248 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:06.249 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:06.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.248+0000 7fbedf7fe640 4 set_mon_vals no callback set 2024-03-25T19:43:06.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.249+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbee001af80 con 0x7fbee813e7a0 2024-03-25T19:43:06.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.249+0000 7fbeed16c640 10 monclient: get_auth_request con 0x7fbecc06de90 auth_method 0 2024-03-25T19:43:06.249 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.249+0000 7fbeddffb640 10 client.4645.objecter ms_handle_connect 0x7fbecc06de90 2024-03-25T19:43:06.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.254+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7fbee0054b30 con 0x7fbee813e7a0 2024-03-25T19:43:06.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.254+0000 7fbeddffb640 10 client.4645.objecter ms_dispatch 0x7fbee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:06.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.254+0000 7fbeddffb640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:06.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.254+0000 7fbeddffb640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:06.254 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.254+0000 7fbeddffb640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T19:43:06.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.256+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbee0050470 con 0x7fbee813e7a0 2024-03-25T19:43:06.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.256+0000 7fbeddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:06.255 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.256+0000 7fbeddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:06.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.490+0000 7fbeef3d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.490+0000 7fbeef3d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.490+0000 7fbeef3d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:06.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.490+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbee80a9ac0 con 0x7fbee813e7a0 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.530+0000 7fbeddffb640 1 -- 172.21.15.29:0/896575879 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7fbee0052570 con 0x7fbee813e7a0 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.530+0000 7fbeddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.530+0000 7fbeddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 10 librados: watch_flush enter 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 10 librados: watch_flush exit 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 10 client.4645.objecter successfully canceled tick 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbecc06de90 legacy=0x7fbecc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 10 monclient: shutdown 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 >> v1:172.21.15.196:6789/0 conn(0x7fbee813e7a0 legacy=0x7fbee81d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 shutdown_connections 2024-03-25T19:43:06.533 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 >> 172.21.15.29:0/896575879 conn(0x7fbee80b7ea0 msgr2=0x7fbee80b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:06.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 shutdown_connections 2024-03-25T19:43:06.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 -- 172.21.15.29:0/896575879 wait complete. 2024-03-25T19:43:06.534 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:06.533+0000 7fbeef3d0640 1 librados: shutdown 2024-03-25T19:43:06.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:11.304 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:41.467 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.316 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:41.479 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.349 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:41.513 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.361 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:41.525 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.369 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:41.533 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.383 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:41.545 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.393 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:41.555 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.399 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:41.563 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:11.419 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:09:41.581 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:13.351 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:13.472 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.472+0000 7f49cfa96640 10 monclient: get_monmap_and_config 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.472+0000 7f49cfa96640 10 monclient: build_initial_monmap 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.472+0000 7f49cfa96640 10 monclient: monmap: 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:13.473981+0000 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:13.473981+0000 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:13.473 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.472+0000 7f49cfa96640 1 Processor -- start 2024-03-25T19:43:13.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 1 -- start start 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient: init 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-b con 0x7f49c80a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-c con 0x7f49c813f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-a con 0x7f49c81430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c8142520 con 0x7f49c81430c0 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80b28a0 con 0x7f49c813f710 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80b2a30 con 0x7f49c80a98f0 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:13.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.473+0000 7f49cfa96640 10 monclient(hunting): authenticate will time out at 6670.227692s 2024-03-25T19:43:13.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.489+0000 7f49ce033640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49c81430c0 0x7f49c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39626/0 (socket says 172.21.15.29:39626) 2024-03-25T19:43:13.490 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.489+0000 7f49ce033640 1 -- 172.21.15.29:0/1707646626 learned_addr learned my addr 172.21.15.29:0/1707646626 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146644455 0 0) 0x7f49c8142520 con 0x7f49c81430c0 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:43:13.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.505+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b4003520 con 0x7f49c81430c0 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099524063 0 0) 0x7f49c80b28a0 con 0x7f49c813f710 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:43:13.509 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.508+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c8142520 con 0x7f49c813f710 2024-03-25T19:43:13.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230811805 0 0) 0x7f49c80b2a30 con 0x7f49c80a98f0 2024-03-25T19:43:13.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.515 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:43:13.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.514+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c80b28a0 con 0x7f49c80a98f0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226288356 0 0) 0x7f49b4003520 con 0x7f49c81430c0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 >> v1:172.21.15.29:6790/0 conn(0x7f49c80a98f0 legacy=0x7f49c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 >> v1:172.21.15.196:6789/0 conn(0x7f49c813f710 legacy=0x7f49c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: _finish_hunting 0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 monclient: found mon.noname-a 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80b44c0 con 0x7f49c81430c0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: _finish_auth 0 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: _check_auth_tickets 2024-03-25T19:43:13.530 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49c0001a10 con 0x7f49c81430c0 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cfa96640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cfa96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.528+0000 7f49cc830640 10 monclient: dump: 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:13.531 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.529+0000 7f49cfa96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49c0001b70 con 0x7f49c81430c0 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 1 -- 172.21.15.29:0/1707646626 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49c0001f80 con 0x7f49c81430c0 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cfa96640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cfa96640 4 set_mon_vals no callback set 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.541 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.540+0000 7f49cc830640 10 monclient: dump: 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:13.542 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.544 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 10 monclient: shutdown 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 1 -- 172.21.15.29:0/1707646626 >> v1:172.21.15.29:6789/0 conn(0x7f49c81430c0 legacy=0x7f49c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 1 -- 172.21.15.29:0/1707646626 shutdown_connections 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 1 -- 172.21.15.29:0/1707646626 >> 172.21.15.29:0/1707646626 conn(0x7f49c80b7ea0 msgr2=0x7f49c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 1 -- 172.21.15.29:0/1707646626 shutdown_connections 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.541+0000 7f49cfa96640 1 -- 172.21.15.29:0/1707646626 wait complete. 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 10 monclient: build_initial_monmap 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 10 monclient: monmap: 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:13.545 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 librados: starting msgr at 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 librados: starting objecter 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 Processor -- start 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 -- start start 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 librados: setting wanted keys 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 1 librados: calling monclient init 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.542+0000 7f49cfa96640 10 monclient: init 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-b con 0x7f49c80a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-a con 0x7f49c813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:13.546 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient(hunting): picked mon.noname-c con 0x7f49c81430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80b7c30 con 0x7f49c813f710 2024-03-25T19:43:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c8137520 con 0x7f49c80a98f0 2024-03-25T19:43:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c81d7ba0 con 0x7f49c81430c0 2024-03-25T19:43:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:13.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.543+0000 7f49cfa96640 10 monclient(hunting): authenticate will time out at 6670.297530s 2024-03-25T19:43:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.549+0000 7f49cd031640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49c813f710 0x7f49c8133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39662/0 (socket says 172.21.15.29:39662) 2024-03-25T19:43:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.549+0000 7f49cd031640 1 -- 172.21.15.29:0/2697163866 learned_addr learned my addr 172.21.15.29:0/2697163866 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.550+0000 7f49be7fc640 10 client.?.objecter ms_handle_connect 0x7f49c813f710 2024-03-25T19:43:13.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.550+0000 7f49be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767956675 0 0) 0x7f49c80b7c30 con 0x7f49c813f710 2024-03-25T19:43:13.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.555 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:43:13.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.555+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac004eb0 con 0x7f49c813f710 2024-03-25T19:43:13.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.557+0000 7f49be7fc640 10 client.?.objecter ms_handle_connect 0x7f49c80a98f0 2024-03-25T19:43:13.558 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.557+0000 7f49be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.565+0000 7f49be7fc640 10 client.?.objecter ms_handle_connect 0x7f49c81430c0 2024-03-25T19:43:13.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.565+0000 7f49be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413985690 0 0) 0x7f49c8137520 con 0x7f49c80a98f0 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:43:13.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.574+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c80b7c30 con 0x7f49c80a98f0 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156650102 0 0) 0x7f49ac004eb0 con 0x7f49c813f710 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c8137520 con 0x7f49c813f710 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49b8001a30 con 0x7f49c813f710 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:13.576 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.577 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.575+0000 7f49be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097712530 0 0) 0x7f49c81d7ba0 con 0x7f49c81430c0 2024-03-25T19:43:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.582 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:43:13.583 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.582+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac004eb0 con 0x7f49c81430c0 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230959306 0 0) 0x7f49c80b7c30 con 0x7f49c80a98f0 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c81d7ba0 con 0x7f49c80a98f0 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49c4001870 con 0x7f49c80a98f0 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.597+0000 7f49be7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:13.598 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:13.599 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:13.600 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:13.600 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1039873738 0 0) 0x7f49c8137520 con 0x7f49c813f710 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 >> v1:172.21.15.29:6790/0 conn(0x7f49c81430c0 legacy=0x7f49c81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 >> v1:172.21.15.196:6789/0 conn(0x7f49c80a98f0 legacy=0x7f49c81d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 1 monclient: found mon.a 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8136950 con 0x7f49c813f710 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:13.606 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 10 monclient: _renew_subs 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c8136ba0 con 0x7f49c813f710 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 10 client.4650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 10 monclient: _renew_subs 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c8137010 con 0x7f49c813f710 2024-03-25T19:43:13.607 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.605+0000 7f49cfa96640 1 librados: init done 2024-03-25T19:43:13.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.606+0000 7f49cfa96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.606+0000 7f49cfa96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.606+0000 7f49cfa96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:13.608 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.606+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c80627b0 con 0x7f49c813f710 2024-03-25T19:43:13.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.612+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b8001cf0 con 0x7f49c813f710 2024-03-25T19:43:13.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.612+0000 7f49be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:13.612 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.612+0000 7f49bffff640 4 set_mon_vals no callback set 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.921+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f49b8003a60 con 0x7f49c813f710 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.921+0000 7f49be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.921+0000 7f49be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.921+0000 7f49be7fc640 10 monclient: dump: 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:13.922 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.922+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f49b8003c40 con 0x7f49c813f710 2024-03-25T19:43:13.923 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.923+0000 7f49cd832640 10 monclient: get_auth_request con 0x7f49ac06df60 auth_method 0 2024-03-25T19:43:13.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.923+0000 7f49be7fc640 10 client.4650.objecter ms_handle_connect 0x7f49ac06df60 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f49b80548c0 con 0x7f49c813f710 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 10 client.4650.objecter ms_dispatch 0x7f49c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 3 client.4650.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 3 client.4650.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 20 client.4650.objecter dump_active .. 0 homeless 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49b80503d0 con 0x7f49c813f710 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:13.930+0000 7f49be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:14.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.153+0000 7f49cfa96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.153+0000 7f49cfa96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.153+0000 7f49cfa96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:14.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.153+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49c8074cc0 con 0x7f49c813f710 2024-03-25T19:43:14.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.174+0000 7f49be7fc640 1 -- 172.21.15.29:0/2697163866 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f49b8052300 con 0x7f49c813f710 2024-03-25T19:43:14.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.174+0000 7f49be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.175 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.174+0000 7f49be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:14.175 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:14.175 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:14.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 10 librados: watch_flush enter 2024-03-25T19:43:14.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 10 librados: watch_flush exit 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 20 client.4650.objecter shutdown clearing up homeless session... 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 10 client.4650.objecter successfully canceled tick 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 >> v1:172.21.15.29:6800/2314944989 conn(0x7f49ac06df60 legacy=0x7f49ac070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 10 monclient: shutdown 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 >> v1:172.21.15.29:6789/0 conn(0x7f49c813f710 legacy=0x7f49c8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 shutdown_connections 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 >> 172.21.15.29:0/2697163866 conn(0x7f49c80b7ea0 msgr2=0x7f49c813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 shutdown_connections 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 -- 172.21.15.29:0/2697163866 wait complete. 2024-03-25T19:43:14.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:14.176+0000 7f49cfa96640 1 librados: shutdown 2024-03-25T19:43:14.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:20.995 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:21.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.113+0000 7f9643e4a640 10 monclient: get_monmap_and_config 2024-03-25T19:43:21.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.113+0000 7f9643e4a640 10 monclient: build_initial_monmap 2024-03-25T19:43:21.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.113+0000 7f9643e4a640 10 monclient: monmap: 2024-03-25T19:43:21.114 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:21.114 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:21.115685+0000 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:21.115685+0000 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:21.115 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.114+0000 7f9643e4a640 1 Processor -- start 2024-03-25T19:43:21.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.114+0000 7f9643e4a640 1 -- start start 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.114+0000 7f9643e4a640 10 monclient: init 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f963c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f963c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f963c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c142520 con 0x7f963c0a98f0 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0b28a0 con 0x7f963c1430c0 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0b2a30 con 0x7f963c13f710 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:21.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.115+0000 7f9643e4a640 10 monclient(hunting): authenticate will time out at 6677.869649s 2024-03-25T19:43:21.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.121+0000 7f96413e5640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f963c13f710 0x7f963c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:49142/0 (socket says 172.21.15.29:49142) 2024-03-25T19:43:21.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.121+0000 7f96413e5640 1 -- 172.21.15.29:0/3971699603 learned_addr learned my addr 172.21.15.29:0/3971699603 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677717534 0 0) 0x7f963c142520 con 0x7f963c0a98f0 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:43:21.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.150+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9628003520 con 0x7f963c0a98f0 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266198953 0 0) 0x7f963c0b2a30 con 0x7f963c13f710 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:43:21.152 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.151+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c142520 con 0x7f963c13f710 2024-03-25T19:43:21.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768824897 0 0) 0x7f963c0b28a0 con 0x7f963c1430c0 2024-03-25T19:43:21.156 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:43:21.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.156+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c0b2a30 con 0x7f963c1430c0 2024-03-25T19:43:21.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376555258 0 0) 0x7f963c142520 con 0x7f963c13f710 2024-03-25T19:43:21.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 >> v1:172.21.15.196:6789/0 conn(0x7f963c1430c0 legacy=0x7f963c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 >> v1:172.21.15.29:6789/0 conn(0x7f963c0a98f0 legacy=0x7f963c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: _finish_hunting 0 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 monclient: found mon.noname-b 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c0b44c0 con 0x7f963c13f710 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: _finish_auth 0 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: _check_auth_tickets 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f962c001670 con 0x7f963c13f710 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9643e4a640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9643e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9640be4640 10 monclient: dump: 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:21.161 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.159+0000 7f9643e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f962c0017d0 con 0x7f963c13f710 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 1 -- 172.21.15.29:0/3971699603 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f962c001be0 con 0x7f963c13f710 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9640be4640 10 monclient: dump: 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:21.178 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9643e4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:21.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.177+0000 7f9643e4a640 4 set_mon_vals no callback set 2024-03-25T19:43:21.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 10 monclient: shutdown 2024-03-25T19:43:21.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 1 -- 172.21.15.29:0/3971699603 >> v1:172.21.15.29:6790/0 conn(0x7f963c13f710 legacy=0x7f963c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 1 -- 172.21.15.29:0/3971699603 shutdown_connections 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 1 -- 172.21.15.29:0/3971699603 >> 172.21.15.29:0/3971699603 conn(0x7f963c0b7ea0 msgr2=0x7f963c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 1 -- 172.21.15.29:0/3971699603 shutdown_connections 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.178+0000 7f9643e4a640 1 -- 172.21.15.29:0/3971699603 wait complete. 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 10 monclient: build_initial_monmap 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 10 monclient: monmap: 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:21.182 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 librados: starting msgr at 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 librados: starting objecter 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 Processor -- start 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 -- start start 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 librados: setting wanted keys 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 1 librados: calling monclient init 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.179+0000 7f9643e4a640 10 monclient: init 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:21.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f963c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f963c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f963c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c136fa0 con 0x7f963c0a98f0 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c13b780 con 0x7f963c13f710 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c1d7ba0 con 0x7f963c1430c0 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:21.184 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.180+0000 7f9643e4a640 10 monclient(hunting): authenticate will time out at 6677.934595s 2024-03-25T19:43:21.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.187+0000 7f96413e5640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f963c13f710 0x7f963c1d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:49624/0 (socket says 172.21.15.29:49624) 2024-03-25T19:43:21.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.187+0000 7f96413e5640 1 -- 172.21.15.29:0/256488346 learned_addr learned my addr 172.21.15.29:0/256488346 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.191+0000 7f96327fc640 10 client.?.objecter ms_handle_connect 0x7f963c13f710 2024-03-25T19:43:21.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.191+0000 7f96327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.200+0000 7f96327fc640 10 client.?.objecter ms_handle_connect 0x7f963c0a98f0 2024-03-25T19:43:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.200+0000 7f96327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.201+0000 7f96327fc640 10 client.?.objecter ms_handle_connect 0x7f963c1430c0 2024-03-25T19:43:21.201 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.201+0000 7f96327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.205 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169119996 0 0) 0x7f963c13b780 con 0x7f963c13f710 2024-03-25T19:43:21.205 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:43:21.206 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.205+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620004eb0 con 0x7f963c13f710 2024-03-25T19:43:21.211 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881928026 0 0) 0x7f963c1d7ba0 con 0x7f963c1430c0 2024-03-25T19:43:21.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:43:21.212 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.211+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c13b780 con 0x7f963c1430c0 2024-03-25T19:43:21.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 176552165 0 0) 0x7f9620004eb0 con 0x7f963c13f710 2024-03-25T19:43:21.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c1d7ba0 con 0x7f963c13f710 2024-03-25T19:43:21.220 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f962c001be0 con 0x7f963c13f710 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 10 monclient(hunting): dump: 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.221 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.220+0000 7f96327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883854905 0 0) 0x7f963c136fa0 con 0x7f963c0a98f0 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.223+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620004eb0 con 0x7f963c0a98f0 2024-03-25T19:43:21.229 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3012936868 0 0) 0x7f963c1d7ba0 con 0x7f963c13f710 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 >> v1:172.21.15.29:6790/0 conn(0x7f963c1430c0 legacy=0x7f963c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 >> v1:172.21.15.29:6789/0 conn(0x7f963c0a98f0 legacy=0x7f963c136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 1 monclient: found mon.b 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c137990 con 0x7f963c13f710 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:21.230 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f96327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 10 monclient: _renew_subs 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c137c40 con 0x7f963c13f710 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 10 monclient: _renew_subs 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c139860 con 0x7f963c13f710 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.228+0000 7f9643e4a640 1 librados: init done 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.229+0000 7f9643e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.229+0000 7f9643e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.229+0000 7f9643e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.229+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c1375f0 con 0x7f963c13f710 2024-03-25T19:43:21.236 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.236+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f962c004a90 con 0x7f963c13f710 2024-03-25T19:43:21.237 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.236+0000 7f96327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:21.237 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.236+0000 7f9633fff640 4 set_mon_vals no callback set 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.509+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f962c004e20 con 0x7f963c13f710 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.509+0000 7f96327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.509+0000 7f96327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.509+0000 7f96327fc640 10 monclient: dump: 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:21.510 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:21.511 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:21.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.510+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f962c005000 con 0x7f963c13f710 2024-03-25T19:43:21.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.511+0000 7f9641be6640 10 monclient: get_auth_request con 0x7f962006dd60 auth_method 0 2024-03-25T19:43:21.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.511+0000 7f96327fc640 10 client.4657.objecter ms_handle_connect 0x7f962006dd60 2024-03-25T19:43:21.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.516+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f962c0549d0 con 0x7f963c13f710 2024-03-25T19:43:21.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.516+0000 7f96327fc640 10 client.4657.objecter ms_dispatch 0x7f963c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:21.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.516+0000 7f96327fc640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:21.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.516+0000 7f96327fc640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:21.517 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.516+0000 7f96327fc640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T19:43:21.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.519+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f962c0504e0 con 0x7f963c13f710 2024-03-25T19:43:21.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.519+0000 7f96327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.519+0000 7f96327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:21.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.744+0000 7f9643e4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.744+0000 7f9643e4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.744+0000 7f9643e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:21.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.744+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f963c0627b0 con 0x7f963c13f710 2024-03-25T19:43:21.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.772+0000 7f96327fc640 1 -- 172.21.15.29:0/256488346 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f962c052410 con 0x7f963c13f710 2024-03-25T19:43:21.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.772+0000 7f96327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.772+0000 7f96327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:21.773 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:21.773 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 10 librados: watch_flush enter 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 10 librados: watch_flush exit 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 10 client.4657.objecter successfully canceled tick 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 >> v1:172.21.15.29:6800/2314944989 conn(0x7f962006dd60 legacy=0x7f9620070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 10 monclient: shutdown 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 >> v1:172.21.15.196:6789/0 conn(0x7f963c13f710 legacy=0x7f963c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 shutdown_connections 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 >> 172.21.15.29:0/256488346 conn(0x7f963c0b7ea0 msgr2=0x7f963c14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 shutdown_connections 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.775+0000 7f9643e4a640 1 -- 172.21.15.29:0/256488346 wait complete. 2024-03-25T19:43:21.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:21.776+0000 7f9643e4a640 1 librados: shutdown 2024-03-25T19:43:21.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:28.594 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:28.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.715+0000 7f517d4a8640 10 monclient: get_monmap_and_config 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.715+0000 7f517d4a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.715+0000 7f517d4a8640 10 monclient: monmap: 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:28.716195+0000 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:28.716195+0000 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:28.715 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:28.716 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:28.716 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 1 Processor -- start 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 1 -- start start 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient: init 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f51780a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f517813f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f51781430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.716+0000 7f517d4a8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5178142520 con 0x7f517813f710 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.717+0000 7f517d4a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780b28a0 con 0x7f51781430c0 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.717+0000 7f517d4a8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780b2a30 con 0x7f51780a98f0 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.717+0000 7f517d4a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:28.717 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.717+0000 7f517d4a8640 10 monclient(hunting): authenticate will time out at 6685.469892s 2024-03-25T19:43:28.729 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.730+0000 7f51777fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f51781430c0 0x7f517814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:38488/0 (socket says 172.21.15.29:38488) 2024-03-25T19:43:28.729 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.730+0000 7f51777fe640 1 -- 172.21.15.29:0/1576467965 learned_addr learned my addr 172.21.15.29:0/1576467965 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368005170 0 0) 0x7f5178142520 con 0x7f517813f710 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:43:28.741 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.741+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160003520 con 0x7f517813f710 2024-03-25T19:43:28.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421464663 0 0) 0x7f51780b28a0 con 0x7f51781430c0 2024-03-25T19:43:28.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:43:28.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.743+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5178142520 con 0x7f51781430c0 2024-03-25T19:43:28.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831243838 0 0) 0x7f51780b2a30 con 0x7f51780a98f0 2024-03-25T19:43:28.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.748 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:43:28.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.749+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51780b28a0 con 0x7f51780a98f0 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1765430231 0 0) 0x7f51780b28a0 con 0x7f51780a98f0 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 >> v1:172.21.15.196:6789/0 conn(0x7f51781430c0 legacy=0x7f517814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 >> v1:172.21.15.29:6789/0 conn(0x7f517813f710 legacy=0x7f51781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 monclient: found mon.noname-b 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:28.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780b44c0 con 0x7f51780a98f0 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f51680015b0 con 0x7f51780a98f0 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f5175ffb640 10 monclient: dump: 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:28.758 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:28.759 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:28.759 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:28.759 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f517d4a8640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:43:28.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.757+0000 7f517d4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:28.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5168001710 con 0x7f51780a98f0 2024-03-25T19:43:28.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:28.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 1 -- 172.21.15.29:0/1576467965 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5168001b60 con 0x7f51780a98f0 2024-03-25T19:43:28.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.772+0000 7f5175ffb640 10 monclient: dump: 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:28.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 4 set_mon_vals no callback set 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 10 monclient: shutdown 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 1 -- 172.21.15.29:0/1576467965 >> v1:172.21.15.29:6790/0 conn(0x7f51780a98f0 legacy=0x7f51780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 1 -- 172.21.15.29:0/1576467965 shutdown_connections 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 1 -- 172.21.15.29:0/1576467965 >> 172.21.15.29:0/1576467965 conn(0x7f51780b7ea0 msgr2=0x7f51780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 1 -- 172.21.15.29:0/1576467965 shutdown_connections 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.773+0000 7f517d4a8640 1 -- 172.21.15.29:0/1576467965 wait complete. 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.774+0000 7f517d4a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.774+0000 7f517d4a8640 10 monclient: monmap: 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:28.775 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 librados: starting msgr at 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 librados: starting objecter 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 Processor -- start 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 -- start start 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 librados: setting wanted keys 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 1 librados: calling monclient init 2024-03-25T19:43:28.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.775+0000 7f517d4a8640 10 monclient: init 2024-03-25T19:43:28.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f51780a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f517813f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f51781430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51781394b0 con 0x7f51780a98f0 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5178139720 con 0x7f51781430c0 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517814e760 con 0x7f517813f710 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:28.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.776+0000 7f517d4a8640 10 monclient(hunting): authenticate will time out at 6685.529447s 2024-03-25T19:43:28.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.784+0000 7f51777fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f51781430c0 0x7f51781e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:38530/0 (socket says 172.21.15.29:38530) 2024-03-25T19:43:28.784 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.784+0000 7f51777fe640 1 -- 172.21.15.29:0/2866323976 learned_addr learned my addr 172.21.15.29:0/2866323976 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:28.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.792+0000 7f515f7fe640 10 client.?.objecter ms_handle_connect 0x7f51781430c0 2024-03-25T19:43:28.792 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.792+0000 7f515f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.795+0000 7f515f7fe640 10 client.?.objecter ms_handle_connect 0x7f517813f710 2024-03-25T19:43:28.795 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.795+0000 7f515f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 client.?.objecter ms_handle_connect 0x7f51780a98f0 2024-03-25T19:43:28.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216414799 0 0) 0x7f5178139720 con 0x7f51781430c0 2024-03-25T19:43:28.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:43:28.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.808+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5164004eb0 con 0x7f51781430c0 2024-03-25T19:43:28.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723686957 0 0) 0x7f517814e760 con 0x7f517813f710 2024-03-25T19:43:28.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:43:28.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.818+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5178139720 con 0x7f517813f710 2024-03-25T19:43:28.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593163598 0 0) 0x7f5164004eb0 con 0x7f51781430c0 2024-03-25T19:43:28.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f517814e760 con 0x7f51781430c0 2024-03-25T19:43:28.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5170001c10 con 0x7f51781430c0 2024-03-25T19:43:28.824 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.824+0000 7f515f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855724730 0 0) 0x7f51781394b0 con 0x7f51780a98f0 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:43:28.828 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.829+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5164004eb0 con 0x7f51780a98f0 2024-03-25T19:43:28.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3125849481 0 0) 0x7f5178139720 con 0x7f517813f710 2024-03-25T19:43:28.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51781394b0 con 0x7f517813f710 2024-03-25T19:43:28.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f516c001950 con 0x7f517813f710 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.841+0000 7f515f7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:28.841 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3820540957 0 0) 0x7f517814e760 con 0x7f51781430c0 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 >> v1:172.21.15.29:6790/0 conn(0x7f517813f710 legacy=0x7f51781db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 >> v1:172.21.15.29:6789/0 conn(0x7f51780a98f0 legacy=0x7f51781da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 1 monclient: found mon.b 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517814e8f0 con 0x7f51781430c0 2024-03-25T19:43:28.848 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f515f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 10 monclient: _renew_subs 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178137610 con 0x7f51781430c0 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 10 monclient: _renew_subs 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178138390 con 0x7f51781430c0 2024-03-25T19:43:28.849 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.848+0000 7f517d4a8640 1 librados: init done 2024-03-25T19:43:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.849+0000 7f517d4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.849+0000 7f517d4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.849+0000 7f517d4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:28.850 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:28.849+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517814f320 con 0x7f51781430c0 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5170001d70 con 0x7f51781430c0 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5170004d00 con 0x7f51781430c0 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f515f7fe640 10 monclient: dump: 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:29.131 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.131+0000 7f51757fa640 4 set_mon_vals no callback set 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.132+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f517001ad50 con 0x7f51781430c0 2024-03-25T19:43:29.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.132+0000 7f5176ffd640 10 monclient: get_auth_request con 0x7f516406df20 auth_method 0 2024-03-25T19:43:29.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.132+0000 7f515f7fe640 10 client.4663.objecter ms_handle_connect 0x7f516406df20 2024-03-25T19:43:29.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.139+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f5170054b20 con 0x7f51781430c0 2024-03-25T19:43:29.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.139+0000 7f515f7fe640 10 client.4663.objecter ms_dispatch 0x7f5178000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:29.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.139+0000 7f515f7fe640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:29.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.139+0000 7f515f7fe640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:29.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.139+0000 7f515f7fe640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:43:29.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.141+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5170050460 con 0x7f51781430c0 2024-03-25T19:43:29.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.141+0000 7f515f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:29.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.141+0000 7f515f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:29.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.375+0000 7f517d4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.375+0000 7f517d4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.375+0000 7f517d4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:29.375 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.375+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51780627b0 con 0x7f51781430c0 2024-03-25T19:43:29.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.406+0000 7f515f7fe640 1 -- 172.21.15.29:0/2866323976 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f5170052560 con 0x7f51781430c0 2024-03-25T19:43:29.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.406+0000 7f515f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.406+0000 7f515f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:29.406 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:29.406 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 10 librados: watch_flush enter 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 10 librados: watch_flush exit 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 >> v1:172.21.15.29:6800/2314944989 conn(0x7f516406df20 legacy=0x7f51640703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 10 monclient: shutdown 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.409+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 >> v1:172.21.15.196:6789/0 conn(0x7f51781430c0 legacy=0x7f51781e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.410+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 shutdown_connections 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.410+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 >> 172.21.15.29:0/2866323976 conn(0x7f51780b7ea0 msgr2=0x7f5178074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.410+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 shutdown_connections 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.410+0000 7f517d4a8640 1 -- 172.21.15.29:0/2866323976 wait complete. 2024-03-25T19:43:29.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:29.410+0000 7f517d4a8640 1 librados: shutdown 2024-03-25T19:43:29.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:31.378 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:01.541 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.380 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:01.543 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.412 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:01.575 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.440 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:01.603 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.441 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:01.604 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.446 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:01.609 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.449 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:01.611 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.465 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:01.629 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.466 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:01.628 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.227 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:36.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.342+0000 7f4f5b3a8640 10 monclient: get_monmap_and_config 2024-03-25T19:43:36.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.342+0000 7f4f5b3a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:36.342 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.342+0000 7f4f5b3a8640 10 monclient: monmap: 2024-03-25T19:43:36.342 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:36.342 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:36.343549+0000 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:36.343549+0000 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:36.343 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.343+0000 7f4f5b3a8640 1 Processor -- start 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.343+0000 7f4f5b3a8640 1 -- start start 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.343+0000 7f4f5b3a8640 10 monclient: init 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4f540a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4f5413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:36.345 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4f541430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:36.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54142520 con 0x7f4f540a98f0 2024-03-25T19:43:36.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540b28a0 con 0x7f4f541430c0 2024-03-25T19:43:36.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540b2a30 con 0x7f4f5413f710 2024-03-25T19:43:36.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:36.346 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.344+0000 7f4f5b3a8640 10 monclient(hunting): authenticate will time out at 6693.097345s 2024-03-25T19:43:36.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.350+0000 7f4f58943640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f4f5413f710 0x7f4f541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:39448/0 (socket says 172.21.15.29:39448) 2024-03-25T19:43:36.350 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.350+0000 7f4f58943640 1 -- 172.21.15.29:0/835051492 learned_addr learned my addr 172.21.15.29:0/835051492 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102762721 0 0) 0x7f4f540b2a30 con 0x7f4f5413f710 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:43:36.363 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.363+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f40003520 con 0x7f4f5413f710 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915514080 0 0) 0x7f4f540b28a0 con 0x7f4f541430c0 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:43:36.374 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.374+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f540b2a30 con 0x7f4f541430c0 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573132299 0 0) 0x7f4f40003520 con 0x7f4f5413f710 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 >> v1:172.21.15.196:6789/0 conn(0x7f4f541430c0 legacy=0x7f4f5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 >> v1:172.21.15.29:6789/0 conn(0x7f4f540a98f0 legacy=0x7f4f540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 1 monclient: found mon.noname-b 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:36.379 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540b44c0 con 0x7f4f5413f710 2024-03-25T19:43:36.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:36.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:36.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:36.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f5b3a8640 5 monclient: authenticate success, global_id 4655 2024-03-25T19:43:36.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.379+0000 7f4f5b3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.381+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f44001650 con 0x7f4f5413f710 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.381+0000 7f4f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.381+0000 7f4f4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.381+0000 7f4f4bfff640 10 monclient: dump: 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.381 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.381+0000 7f4f5b3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f440018a0 con 0x7f4f5413f710 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f4bfff640 1 -- 172.21.15.29:0/835051492 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f44001c30 con 0x7f4f5413f710 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f5b3a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f5b3a8640 4 set_mon_vals no callback set 2024-03-25T19:43:36.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.393+0000 7f4f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f4bfff640 10 monclient: dump: 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.394 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.395 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f5b3a8640 10 monclient: shutdown 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f5b3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/835051492 >> v1:172.21.15.29:6790/0 conn(0x7f4f5413f710 legacy=0x7f4f541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/835051492 shutdown_connections 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.394+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/835051492 >> 172.21.15.29:0/835051492 conn(0x7f4f540b7ea0 msgr2=0x7f4f540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.395+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/835051492 shutdown_connections 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.395+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/835051492 wait complete. 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.395+0000 7f4f5b3a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.395+0000 7f4f5b3a8640 10 monclient: monmap: 2024-03-25T19:43:36.397 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 librados: starting msgr at 2024-03-25T19:43:36.398 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 librados: starting objecter 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 Processor -- start 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 -- start start 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 librados: setting wanted keys 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 librados: calling monclient init 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient: init 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4f540a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4f5413f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4f541430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540b7cc0 con 0x7f4f540a98f0 2024-03-25T19:43:36.399 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f541dc070 con 0x7f4f541430c0 2024-03-25T19:43:36.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54137140 con 0x7f4f5413f710 2024-03-25T19:43:36.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:36.400 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.396+0000 7f4f5b3a8640 10 monclient(hunting): authenticate will time out at 6693.149858s 2024-03-25T19:43:36.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.407+0000 7f4f59144640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f540a98f0 0x7f4f54137840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49724/0 (socket says 172.21.15.29:49724) 2024-03-25T19:43:36.406 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.407+0000 7f4f59144640 1 -- 172.21.15.29:0/2900057841 learned_addr learned my addr 172.21.15.29:0/2900057841 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:36.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.409+0000 7f4f49ffb640 10 client.?.objecter ms_handle_connect 0x7f4f540a98f0 2024-03-25T19:43:36.408 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.409+0000 7f4f49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.417+0000 7f4f49ffb640 10 client.?.objecter ms_handle_connect 0x7f4f541430c0 2024-03-25T19:43:36.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.417+0000 7f4f49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.422+0000 7f4f49ffb640 10 client.?.objecter ms_handle_connect 0x7f4f5413f710 2024-03-25T19:43:36.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.422+0000 7f4f49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211426299 0 0) 0x7f4f540b7cc0 con 0x7f4f540a98f0 2024-03-25T19:43:36.434 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:43:36.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.434+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38004eb0 con 0x7f4f540a98f0 2024-03-25T19:43:36.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522999189 0 0) 0x7f4f54137140 con 0x7f4f5413f710 2024-03-25T19:43:36.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.437 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:43:36.438 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.437+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f540b7cc0 con 0x7f4f5413f710 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077507402 0 0) 0x7f4f541dc070 con 0x7f4f541430c0 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:43:36.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.447+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54137140 con 0x7f4f541430c0 2024-03-25T19:43:36.452 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 588299359 0 0) 0x7f4f38004eb0 con 0x7f4f540a98f0 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f541dc070 con 0x7f4f540a98f0 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f50001a30 con 0x7f4f540a98f0 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.453 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.454 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.454 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.453+0000 7f4f49ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:36.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898956281 0 0) 0x7f4f540b7cc0 con 0x7f4f5413f710 2024-03-25T19:43:36.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f38004eb0 con 0x7f4f5413f710 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f44001bb0 con 0x7f4f5413f710 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.455+0000 7f4f49ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.456 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1543490285 0 0) 0x7f4f54137140 con 0x7f4f541430c0 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f540b7cc0 con 0x7f4f541430c0 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f4c001a30 con 0x7f4f541430c0 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.465+0000 7f4f49ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.465 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.466 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2984595890 0 0) 0x7f4f38004eb0 con 0x7f4f5413f710 2024-03-25T19:43:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 >> v1:172.21.15.196:6789/0 conn(0x7f4f541430c0 legacy=0x7f4f541db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 >> v1:172.21.15.29:6789/0 conn(0x7f4f540a98f0 legacy=0x7f4f54137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:36.482 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 1 monclient: found mon.c 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54137420 con 0x7f4f5413f710 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 10 monclient: _renew_subs 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f54137f50 con 0x7f4f5413f710 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 10 monclient: _renew_subs 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f54136190 con 0x7f4f5413f710 2024-03-25T19:43:36.483 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.481+0000 7f4f5b3a8640 1 librados: init done 2024-03-25T19:43:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.482+0000 7f4f5b3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.482+0000 7f4f5b3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.482+0000 7f4f5b3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:43:36.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.482+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f540b7b30 con 0x7f4f5413f710 2024-03-25T19:43:36.804 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f44001d10 con 0x7f4f5413f710 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f4f44004bf0 con 0x7f4f5413f710 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f49ffb640 10 monclient: dump: 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.805 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.805+0000 7f4f4b7fe640 4 set_mon_vals no callback set 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.806+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f4f4401ac00 con 0x7f4f5413f710 2024-03-25T19:43:36.806 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.806+0000 7f4f59144640 10 monclient: get_auth_request con 0x7f4f3806e160 auth_method 0 2024-03-25T19:43:36.807 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.806+0000 7f4f49ffb640 10 client.4658.objecter ms_handle_connect 0x7f4f3806e160 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f4f440547e0 con 0x7f4f5413f710 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 10 client.4658.objecter ms_dispatch 0x7f4f54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f440502f0 con 0x7f4f5413f710 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.811 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:36.811+0000 7f4f49ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:37.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.038+0000 7f4f5b3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.038+0000 7f4f5b3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.038+0000 7f4f5b3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:43:37.038 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.038+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f540627b0 con 0x7f4f5413f710 2024-03-25T19:43:37.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.088+0000 7f4f49ffb640 1 -- 172.21.15.29:0/2900057841 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f4f44052220 con 0x7f4f5413f710 2024-03-25T19:43:37.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.089+0000 7f4f49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.088 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.089+0000 7f4f49ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:37.089 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:37.089 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:37.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 10 librados: watch_flush enter 2024-03-25T19:43:37.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 10 librados: watch_flush exit 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 10 client.4658.objecter successfully canceled tick 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 >> v1:172.21.15.29:6800/2314944989 conn(0x7f4f3806e160 legacy=0x7f4f38070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 10 monclient: shutdown 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 >> v1:172.21.15.29:6790/0 conn(0x7f4f5413f710 legacy=0x7f4f541db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 shutdown_connections 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 >> 172.21.15.29:0/2900057841 conn(0x7f4f540b7ea0 msgr2=0x7f4f54074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.091+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 shutdown_connections 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.092+0000 7f4f5b3a8640 1 -- 172.21.15.29:0/2900057841 wait complete. 2024-03-25T19:43:37.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:37.092+0000 7f4f5b3a8640 1 librados: shutdown 2024-03-25T19:43:37.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:43.910 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:44.026 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.027+0000 7f9ab79b5640 10 monclient: get_monmap_and_config 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.027+0000 7f9ab79b5640 10 monclient: build_initial_monmap 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.027+0000 7f9ab79b5640 10 monclient: monmap: 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:44.028080+0000 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:44.028080+0000 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.027+0000 7f9ab79b5640 1 Processor -- start 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 1 -- start start 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient: init 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab00b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab00b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00af940 con 0x7f9ab00b4d60 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00b0a70 con 0x7f9ab00b1730 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00b0d50 con 0x7f9ab014be90 2024-03-25T19:43:44.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:44.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.028+0000 7f9ab79b5640 10 monclient(hunting): authenticate will time out at 6700.781696s 2024-03-25T19:43:44.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.035+0000 7f9ab5751640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9ab00b4d60 0x7f9ab013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54258/0 (socket says 172.21.15.29:54258) 2024-03-25T19:43:44.034 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.035+0000 7f9ab5751640 1 -- 172.21.15.29:0/3288740265 learned_addr learned my addr 172.21.15.29:0/3288740265 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.049+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348449059 0 0) 0x7f9ab00b0a70 con 0x7f9ab00b1730 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.050+0000 7f9aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.050+0000 7f9aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.050+0000 7f9aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.050+0000 7f9aa7fff640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:43:44.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.050+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a9c003520 con 0x7f9ab00b1730 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142149468 0 0) 0x7f9ab00af940 con 0x7f9ab00b4d60 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab00b0a70 con 0x7f9ab00b4d60 2024-03-25T19:43:44.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526607061 0 0) 0x7f9ab00b0d50 con 0x7f9ab014be90 2024-03-25T19:43:44.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:43:44.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.056+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab00af940 con 0x7f9ab014be90 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637133776 0 0) 0x7f9ab00af940 con 0x7f9ab014be90 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 >> v1:172.21.15.196:6789/0 conn(0x7f9ab00b1730 legacy=0x7f9ab00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 >> v1:172.21.15.29:6789/0 conn(0x7f9ab00b4d60 legacy=0x7f9ab013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 monclient: found mon.noname-b 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00b1030 con 0x7f9ab014be90 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:44.065 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9aa8001930 con 0x7f9ab014be90 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9aa7fff640 10 monclient: dump: 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:44.066 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:44.067 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:44.067 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9ab79b5640 5 monclient: authenticate success, global_id 4661 2024-03-25T19:43:44.067 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.065+0000 7f9ab79b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:44.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa8001a90 con 0x7f9ab014be90 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 1 -- 172.21.15.29:0/3288740265 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9aa8001ee0 con 0x7f9ab014be90 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9ab79b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9ab79b5640 4 set_mon_vals no callback set 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.076+0000 7f9aa7fff640 10 monclient: dump: 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.076 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.077 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:44.077 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:44.077 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:44.077 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 10 monclient: shutdown 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 1 -- 172.21.15.29:0/3288740265 >> v1:172.21.15.29:6790/0 conn(0x7f9ab014be90 legacy=0x7f9ab014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 1 -- 172.21.15.29:0/3288740265 shutdown_connections 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 1 -- 172.21.15.29:0/3288740265 >> 172.21.15.29:0/3288740265 conn(0x7f9ab00617a0 msgr2=0x7f9ab0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 1 -- 172.21.15.29:0/3288740265 shutdown_connections 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 1 -- 172.21.15.29:0/3288740265 wait complete. 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 10 monclient: build_initial_monmap 2024-03-25T19:43:44.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.077+0000 7f9ab79b5640 10 monclient: monmap: 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 librados: starting msgr at 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 librados: starting objecter 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 Processor -- start 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 -- start start 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 librados: setting wanted keys 2024-03-25T19:43:44.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 1 librados: calling monclient init 2024-03-25T19:43:44.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.078+0000 7f9ab79b5640 10 monclient: init 2024-03-25T19:43:44.080 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab00b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab00b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab014be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab01422c0 con 0x7f9ab014be90 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab0143220 con 0x7f9ab00b1730 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab01d7870 con 0x7f9ab00b4d60 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:44.081 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.079+0000 7f9ab79b5640 10 monclient(hunting): authenticate will time out at 6700.832211s 2024-03-25T19:43:44.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.083+0000 7f9ab5751640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9ab00b1730 0x7f9ab0141bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:32844/0 (socket says 172.21.15.29:32844) 2024-03-25T19:43:44.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.083+0000 7f9ab5751640 1 -- 172.21.15.29:0/2613539420 learned_addr learned my addr 172.21.15.29:0/2613539420 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:44.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.084+0000 7f9aa5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ab00b1730 2024-03-25T19:43:44.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.084+0000 7f9aa5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.092+0000 7f9aa5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ab00b4d60 2024-03-25T19:43:44.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.092+0000 7f9aa5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073539933 0 0) 0x7f9ab0143220 con 0x7f9ab00b1730 2024-03-25T19:43:44.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:43:44.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a94004eb0 con 0x7f9ab00b1730 2024-03-25T19:43:44.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 client.?.objecter ms_handle_connect 0x7f9ab014be90 2024-03-25T19:43:44.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.093+0000 7f9aa5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.110+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669745137 0 0) 0x7f9a94004eb0 con 0x7f9ab00b1730 2024-03-25T19:43:44.110 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.110+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab0143220 con 0x7f9ab00b1730 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027877074 0 0) 0x7f9ab01422c0 con 0x7f9ab014be90 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:43:44.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.114+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a94004eb0 con 0x7f9ab014be90 2024-03-25T19:43:44.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.117+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9aac0019f0 con 0x7f9ab00b1730 2024-03-25T19:43:44.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.117+0000 7f9aa5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.117+0000 7f9aa5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.117+0000 7f9aa5ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.117 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.117+0000 7f9aa5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717122743 0 0) 0x7f9ab01d7870 con 0x7f9ab00b4d60 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:43:44.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.118+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab01422c0 con 0x7f9ab00b4d60 2024-03-25T19:43:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635500156 0 0) 0x7f9a94004eb0 con 0x7f9ab014be90 2024-03-25T19:43:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab01d7870 con 0x7f9ab014be90 2024-03-25T19:43:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9aa8001da0 con 0x7f9ab014be90 2024-03-25T19:43:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.137 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116145588 0 0) 0x7f9ab0143220 con 0x7f9ab00b1730 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 >> v1:172.21.15.29:6790/0 conn(0x7f9ab00b4d60 legacy=0x7f9ab01d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 >> v1:172.21.15.29:6789/0 conn(0x7f9ab014be90 legacy=0x7f9ab0132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 monclient: found mon.b 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab013d490 con 0x7f9ab00b1730 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9aa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 10 monclient: _renew_subs 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab0140ea0 con 0x7f9ab00b1730 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 10 monclient: _renew_subs 2024-03-25T19:43:44.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab01d8680 con 0x7f9ab00b1730 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.137+0000 7f9ab79b5640 1 librados: init done 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.138+0000 7f9ab79b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.138+0000 7f9ab79b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.138+0000 7f9ab79b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:44.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.138+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0141240 con 0x7f9ab00b1730 2024-03-25T19:43:44.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aac0047e0 con 0x7f9ab00b1730 2024-03-25T19:43:44.410 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9aac004bf0 con 0x7f9ab00b1730 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa5ffb640 10 monclient: dump: 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:44.411 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.410+0000 7f9aa77fe640 4 set_mon_vals no callback set 2024-03-25T19:43:44.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.411+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9aac004e10 con 0x7f9ab00b1730 2024-03-25T19:43:44.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.412+0000 7f9ab4f50640 10 monclient: get_auth_request con 0x7f9a9406dfb0 auth_method 0 2024-03-25T19:43:44.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.412+0000 7f9aa5ffb640 10 client.4675.objecter ms_handle_connect 0x7f9a9406dfb0 2024-03-25T19:43:44.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.417+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110308765 0 0) 0x7f9aac054930 con 0x7f9ab00b1730 2024-03-25T19:43:44.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.417+0000 7f9aa5ffb640 10 client.4675.objecter ms_dispatch 0x7f9ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:44.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.417+0000 7f9aa5ffb640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:44.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.417+0000 7f9aa5ffb640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:44.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.417+0000 7f9aa5ffb640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:43:44.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.418+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aac050440 con 0x7f9ab00b1730 2024-03-25T19:43:44.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.418+0000 7f9aa5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.418+0000 7f9aa5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:44.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.659+0000 7f9ab79b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.659+0000 7f9ab79b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.659+0000 7f9ab79b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:44.659 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.659+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ab01d7a00 con 0x7f9ab00b1730 2024-03-25T19:43:44.739 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.740+0000 7f9aa5ffb640 1 -- 172.21.15.29:0/2613539420 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3778100067) 0x7f9aac052370 con 0x7f9ab00b1730 2024-03-25T19:43:44.739 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.740+0000 7f9aa5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.740+0000 7f9aa5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:44.740 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:44.740 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:33:52.661704+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:33:52.661704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:44.742 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 10 librados: watch_flush enter 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 10 librados: watch_flush exit 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 >> v1:172.21.15.29:6800/2314944989 conn(0x7f9a9406dfb0 legacy=0x7f9a94070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 10 monclient: shutdown 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.742+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 >> v1:172.21.15.196:6789/0 conn(0x7f9ab00b1730 legacy=0x7f9ab0141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.743+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 shutdown_connections 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.743+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 >> 172.21.15.29:0/2613539420 conn(0x7f9ab00617a0 msgr2=0x7f9ab00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.743+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 shutdown_connections 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.743+0000 7f9ab79b5640 1 -- 172.21.15.29:0/2613539420 wait complete. 2024-03-25T19:43:44.743 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:44.743+0000 7f9ab79b5640 1 librados: shutdown 2024-03-25T19:43:44.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:51.560 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:51.628 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:21.791 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.632 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:21.796 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.636 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:21.799 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.640 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:21.802 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.651 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:21.815 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.653 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:21.817 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.664+0000 7f70b977b640 10 monclient: get_monmap_and_config 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.664+0000 7f70b977b640 10 monclient: build_initial_monmap 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.664+0000 7f70b977b640 10 monclient: monmap: 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:51.665397+0000 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:51.665397+0000 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:51.664 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:51.665 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:51.665 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 1 Processor -- start 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 1 -- start start 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient: init 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-b con 0x7f70b40b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-c con 0x7f70b40b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-a con 0x7f70b414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40af940 con 0x7f70b414be90 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40b0a70 con 0x7f70b40b1730 2024-03-25T19:43:51.666 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40b0d50 con 0x7f70b40b4d60 2024-03-25T19:43:51.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:51.667 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.665+0000 7f70b977b640 10 monclient(hunting): authenticate will time out at 6708.418770s 2024-03-25T19:43:51.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.689+0000 7f70b27fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f70b40b1730 0x7f70b40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:59396/0 (socket says 172.21.15.29:59396) 2024-03-25T19:43:51.688 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.689+0000 7f70b27fc640 1 -- 172.21.15.29:0/2005697186 learned_addr learned my addr 172.21.15.29:0/2005697186 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:51.696 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:21.858 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.710 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:21.874 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.720 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988808206 0 0) 0x7f70b40b0d50 con 0x7f70b40b4d60 2024-03-25T19:43:51.720 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.720 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.720 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:43:51.721 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.721+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c003520 con 0x7f70b40b4d60 2024-03-25T19:43:51.734 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990462053 0 0) 0x7f709c003520 con 0x7f70b40b4d60 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 >> v1:172.21.15.196:6789/0 conn(0x7f70b40b1730 legacy=0x7f70b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 >> v1:172.21.15.29:6789/0 conn(0x7f70b414be90 legacy=0x7f70b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 monclient: found mon.noname-b 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40b1030 con 0x7f70b40b4d60 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f70a4001690 con 0x7f70b40b4d60 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b977b640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.735 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b1ffb640 10 monclient: dump: 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.734+0000 7f70b977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:51.741 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:21.904 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:51.856 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70a4001860 con 0x7f70b40b4d60 2024-03-25T19:43:51.856 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:51.856 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 1 -- 172.21.15.29:0/2005697186 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f70a4001cb0 con 0x7f70b40b4d60 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.856+0000 7f70b1ffb640 10 monclient: dump: 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:51.858 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 4 set_mon_vals no callback set 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 10 monclient: shutdown 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 1 -- 172.21.15.29:0/2005697186 >> v1:172.21.15.29:6790/0 conn(0x7f70b40b4d60 legacy=0x7f70b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 1 -- 172.21.15.29:0/2005697186 shutdown_connections 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 1 -- 172.21.15.29:0/2005697186 >> 172.21.15.29:0/2005697186 conn(0x7f70b40617a0 msgr2=0x7f70b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.857+0000 7f70b977b640 1 -- 172.21.15.29:0/2005697186 shutdown_connections 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.858+0000 7f70b977b640 1 -- 172.21.15.29:0/2005697186 wait complete. 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.859+0000 7f70b977b640 10 monclient: build_initial_monmap 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.859+0000 7f70b977b640 10 monclient: monmap: 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:51.859 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.859+0000 7f70b977b640 1 librados: starting msgr at 2024-03-25T19:43:51.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.859+0000 7f70b977b640 1 librados: starting objecter 2024-03-25T19:43:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 1 Processor -- start 2024-03-25T19:43:51.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 1 -- start start 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 1 librados: setting wanted keys 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 1 librados: calling monclient init 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 10 monclient: init 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-b con 0x7f70b40b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.860+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-c con 0x7f70b40b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 10 monclient(hunting): picked mon.noname-a con 0x7f70b414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b4142bd0 con 0x7f70b414be90 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b4142880 con 0x7f70b40b1730 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b4140e30 con 0x7f70b40b4d60 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:51.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.861+0000 7f70b977b640 10 monclient(hunting): authenticate will time out at 6708.613979s 2024-03-25T19:43:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.862+0000 7f70b2ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f70b40b1730 0x7f70b41d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:59468/0 (socket says 172.21.15.29:59468) 2024-03-25T19:43:51.863 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.862+0000 7f70b2ffd640 1 -- 172.21.15.29:0/4074094879 learned_addr learned my addr 172.21.15.29:0/4074094879 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:51.864 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.865+0000 7f709b7fe640 10 client.?.objecter ms_handle_connect 0x7f70b40b1730 2024-03-25T19:43:51.864 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.865+0000 7f709b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.869+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753448174 0 0) 0x7f70b4142880 con 0x7f70b40b1730 2024-03-25T19:43:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.870+0000 7f709b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.870+0000 7f709b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.870+0000 7f709b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.869 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.870+0000 7f709b7fe640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:43:51.870 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.870+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a0004eb0 con 0x7f70b40b1730 2024-03-25T19:43:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.872+0000 7f709b7fe640 10 client.?.objecter ms_handle_connect 0x7f70b414be90 2024-03-25T19:43:51.871 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.872+0000 7f709b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.880+0000 7f709b7fe640 10 client.?.objecter ms_handle_connect 0x7f70b40b4d60 2024-03-25T19:43:51.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.880+0000 7f709b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308496667 0 0) 0x7f70b4142bd0 con 0x7f70b414be90 2024-03-25T19:43:51.880 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:43:51.881 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.881+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4142880 con 0x7f70b414be90 2024-03-25T19:43:51.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455915285 0 0) 0x7f70b4140e30 con 0x7f70b40b4d60 2024-03-25T19:43:51.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:43:51.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.888+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4142bd0 con 0x7f70b40b4d60 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032583136 0 0) 0x7f70a0004eb0 con 0x7f70b40b1730 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b4140e30 con 0x7f70b40b1730 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f70a4001c30 con 0x7f70b40b1730 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:51.891 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.892 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.891+0000 7f709b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054629749 0 0) 0x7f70b4142bd0 con 0x7f70b40b4d60 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a0004eb0 con 0x7f70b40b4d60 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f70a8001860 con 0x7f70b40b4d60 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.905+0000 7f709b7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.905 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:51.906 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:51.906 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:51.906 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:51.906 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:51.907 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4230699442 0 0) 0x7f70b4140e30 con 0x7f70b40b1730 2024-03-25T19:43:51.907 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 >> v1:172.21.15.29:6790/0 conn(0x7f70b40b4d60 legacy=0x7f70b4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.907 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 >> v1:172.21.15.29:6789/0 conn(0x7f70b414be90 legacy=0x7f70b41430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.907 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:51.907 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 1 monclient: found mon.b 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b4140fc0 con 0x7f70b40b1730 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f709b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 10 monclient: _renew_subs 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b41385c0 con 0x7f70b40b1730 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 10 monclient: _renew_subs 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b4137200 con 0x7f70b40b1730 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.906+0000 7f70b977b640 1 librados: init done 2024-03-25T19:43:51.908 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.907+0000 7f70b977b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:51.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.907+0000 7f70b977b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:51.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.907+0000 7f70b977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:51.909 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:51.907+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b4142a10 con 0x7f70b40b1730 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70a4003960 con 0x7f70b40b1730 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f70a4003d70 con 0x7f70b40b1730 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:52.189 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f709b7fe640 10 monclient: dump: 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:52.190 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.189+0000 7f70b17fa640 4 set_mon_vals no callback set 2024-03-25T19:43:52.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.190+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f70a4003f90 con 0x7f70b40b1730 2024-03-25T19:43:52.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.190+0000 7f70b27fc640 10 monclient: get_auth_request con 0x7f70a006df60 auth_method 0 2024-03-25T19:43:52.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.191+0000 7f709b7fe640 10 client.4678.objecter ms_handle_connect 0x7f70a006df60 2024-03-25T19:43:52.196 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f70a4054900 con 0x7f70b40b1730 2024-03-25T19:43:52.196 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 10 client.4678.objecter ms_dispatch 0x7f70b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 3 client.4678.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 3 client.4678.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70a4050410 con 0x7f70b40b1730 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:52.197 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.197+0000 7f709b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:52.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.455+0000 7f70b977b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.455+0000 7f70b977b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.455+0000 7f70b977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:43:52.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.455+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70b4142d60 con 0x7f70b40b1730 2024-03-25T19:43:52.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.486+0000 7f709b7fe640 1 -- 172.21.15.29:0/4074094879 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f70a4052340 con 0x7f70b40b1730 2024-03-25T19:43:52.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.486+0000 7f709b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.486+0000 7f709b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:52.486 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:43:52.486 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 10 librados: watch_flush enter 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 10 librados: watch_flush exit 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 10 client.4678.objecter successfully canceled tick 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 >> v1:172.21.15.29:6800/2314944989 conn(0x7f70a006df60 legacy=0x7f70a0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 10 monclient: shutdown 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 >> v1:172.21.15.196:6789/0 conn(0x7f70b40b1730 legacy=0x7f70b41d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 shutdown_connections 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 >> 172.21.15.29:0/4074094879 conn(0x7f70b40617a0 msgr2=0x7f70b413c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 shutdown_connections 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 -- 172.21.15.29:0/4074094879 wait complete. 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:52.489+0000 7f70b977b640 1 librados: shutdown 2024-03-25T19:43:52.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:59.307 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:59.420 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.420+0000 7f0b34108640 10 monclient: get_monmap_and_config 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.420+0000 7f0b34108640 10 monclient: build_initial_monmap 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.421+0000 7f0b34108640 10 monclient: monmap: 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:43:59.421788+0000 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:43:59.421788+0000 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:43:59.421 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.421+0000 7f0b34108640 1 Processor -- start 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 1 -- start start 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient: init 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c0a98f0 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c142520 con 0x7f0b2c1430c0 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.422+0000 7f0b34108640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0b28a0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.423 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.423+0000 7f0b34108640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0b2a30 con 0x7f0b2c13f710 2024-03-25T19:43:59.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.423+0000 7f0b34108640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:59.424 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.423+0000 7f0b34108640 10 monclient(hunting): authenticate will time out at 6716.175934s 2024-03-25T19:43:59.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.430+0000 7f0b316a3640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f0b2c13f710 0x7f0b2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:48182/0 (socket says 172.21.15.29:48182) 2024-03-25T19:43:59.430 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.430+0000 7f0b316a3640 1 -- 172.21.15.29:0/2317972890 learned_addr learned my addr 172.21.15.29:0/2317972890 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:59.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959055920 0 0) 0x7f0b2c0b2a30 con 0x7f0b2c13f710 2024-03-25T19:43:59.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:43:59.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.442+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18003520 con 0x7f0b2c13f710 2024-03-25T19:43:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445600034 0 0) 0x7f0b2c0b28a0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.459 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:43:59.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.459+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c0b2a30 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183406301 0 0) 0x7f0b18003520 con 0x7f0b2c13f710 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 >> v1:172.21.15.196:6789/0 conn(0x7f0b2c0a98f0 legacy=0x7f0b2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 >> v1:172.21.15.29:6789/0 conn(0x7f0b2c1430c0 legacy=0x7f0b2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: _finish_hunting 0 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 monclient: found mon.noname-b 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:43:59.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0b44c0 con 0x7f0b2c13f710 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: _finish_auth 0 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: _check_auth_tickets 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b1c001690 con 0x7f0b2c13f710 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b30ea2640 10 monclient: dump: 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.464 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:59.465 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:59.465 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:59.465 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b34108640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:43:59.465 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.463+0000 7f0b34108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c0017f0 con 0x7f0b2c13f710 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 1 -- 172.21.15.29:0/2317972890 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b1c001c00 con 0x7f0b2c13f710 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b30ea2640 10 monclient: dump: 2024-03-25T19:43:59.474 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b34108640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:59.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.474+0000 7f0b34108640 4 set_mon_vals no callback set 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.475+0000 7f0b34108640 10 monclient: shutdown 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.475+0000 7f0b34108640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.475+0000 7f0b34108640 1 -- 172.21.15.29:0/2317972890 >> v1:172.21.15.29:6790/0 conn(0x7f0b2c13f710 legacy=0x7f0b2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.475+0000 7f0b34108640 1 -- 172.21.15.29:0/2317972890 shutdown_connections 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.475+0000 7f0b34108640 1 -- 172.21.15.29:0/2317972890 >> 172.21.15.29:0/2317972890 conn(0x7f0b2c0b7ea0 msgr2=0x7f0b2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.476+0000 7f0b34108640 1 -- 172.21.15.29:0/2317972890 shutdown_connections 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.476+0000 7f0b34108640 1 -- 172.21.15.29:0/2317972890 wait complete. 2024-03-25T19:43:59.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.476+0000 7f0b34108640 10 monclient: build_initial_monmap 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.476+0000 7f0b34108640 10 monclient: monmap: 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 librados: starting msgr at 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 librados: starting objecter 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 Processor -- start 2024-03-25T19:43:59.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 -- start start 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 librados: setting wanted keys 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 librados: calling monclient init 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient: init 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c137720 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c1dc070 con 0x7f0b2c13f710 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c139140 con 0x7f0b2c1430c0 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:59.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.477+0000 7f0b34108640 10 monclient(hunting): authenticate will time out at 6716.230790s 2024-03-25T19:43:59.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.485+0000 7f0b31ea4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b2c0a98f0 0x7f0b2c137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43348/0 (socket says 172.21.15.29:43348) 2024-03-25T19:43:59.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.485+0000 7f0b31ea4640 1 -- 172.21.15.29:0/211666864 learned_addr learned my addr 172.21.15.29:0/211666864 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:43:59.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.486+0000 7f0b227fc640 10 client.?.objecter ms_handle_connect 0x7f0b2c0a98f0 2024-03-25T19:43:59.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.486+0000 7f0b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.489+0000 7f0b227fc640 10 client.?.objecter ms_handle_connect 0x7f0b2c1430c0 2024-03-25T19:43:59.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.489+0000 7f0b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109085705 0 0) 0x7f0b2c137720 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:43:59.491 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.491+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10004eb0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.492+0000 7f0b227fc640 10 client.?.objecter ms_handle_connect 0x7f0b2c13f710 2024-03-25T19:43:59.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.492+0000 7f0b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721591907 0 0) 0x7f0b2c139140 con 0x7f0b2c1430c0 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:43:59.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.497+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c137720 con 0x7f0b2c1430c0 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2703329567 0 0) 0x7f0b10004eb0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2c139140 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b28001a10 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 10 monclient(hunting): dump: 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.502 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.503 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.502+0000 7f0b227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.506+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404556789 0 0) 0x7f0b2c1dc070 con 0x7f0b2c13f710 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.506+0000 7f0b227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.506+0000 7f0b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.506+0000 7f0b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.507+0000 7f0b227fc640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:43:59.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.507+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10004eb0 con 0x7f0b2c13f710 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973733892 0 0) 0x7f0b2c137720 con 0x7f0b2c1430c0 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2c1dc070 con 0x7f0b2c1430c0 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b24001b10 con 0x7f0b2c1430c0 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.516+0000 7f0b227fc640 10 monclient(hunting): dump: 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:59.516 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:59.517 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:59.517 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4008555284 0 0) 0x7f0b2c139140 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 >> v1:172.21.15.29:6790/0 conn(0x7f0b2c1430c0 legacy=0x7f0b2c1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 >> v1:172.21.15.196:6789/0 conn(0x7f0b2c13f710 legacy=0x7f0b2c1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 1 monclient: found mon.a 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c139420 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:59.519 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 10 monclient: _renew_subs 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c136dc0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 10 monclient: _renew_subs 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c139840 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.520 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.519+0000 7f0b34108640 1 librados: init done 2024-03-25T19:43:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.520+0000 7f0b34108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.520+0000 7f0b34108640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.520+0000 7f0b34108640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:43:59.521 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.520+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c14f2d0 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b28001c50 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f0b28003a60 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 10 monclient: dump: 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:43:59.843 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:43:59.844 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:43:59.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b23fff640 4 set_mon_vals no callback set 2024-03-25T19:43:59.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.843+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f0b28003c80 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.844+0000 7f0b316a3640 10 monclient: get_auth_request con 0x7f0b1006dfb0 auth_method 0 2024-03-25T19:43:59.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.844+0000 7f0b227fc640 10 client.4683.objecter ms_handle_connect 0x7f0b1006dfb0 2024-03-25T19:43:59.854 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f0b28054860 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 10 client.4683.objecter ms_dispatch 0x7f0b2c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b28050370 con 0x7f0b2c0a98f0 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:43:59.855+0000 7f0b227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.100+0000 7f0b34108640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.100+0000 7f0b34108640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.100+0000 7f0b34108640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:00.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.100+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b2c136f50 con 0x7f0b2c0a98f0 2024-03-25T19:44:00.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.146+0000 7f0b227fc640 1 -- 172.21.15.29:0/211666864 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f0b280522a0 con 0x7f0b2c0a98f0 2024-03-25T19:44:00.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.146+0000 7f0b227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.146+0000 7f0b227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:00.146 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:00.146 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 10 librados: watch_flush enter 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 10 librados: watch_flush exit 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 10 client.4683.objecter successfully canceled tick 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 >> v1:172.21.15.29:6800/2314944989 conn(0x7f0b1006dfb0 legacy=0x7f0b10070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.148+0000 7f0b34108640 10 monclient: shutdown 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 >> v1:172.21.15.29:6789/0 conn(0x7f0b2c0a98f0 legacy=0x7f0b2c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 shutdown_connections 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 >> 172.21.15.29:0/211666864 conn(0x7f0b2c0b7ea0 msgr2=0x7f0b2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 shutdown_connections 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 -- 172.21.15.29:0/211666864 wait complete. 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:00.149+0000 7f0b34108640 1 librados: shutdown 2024-03-25T19:44:00.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:06.967 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:07.083 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.084+0000 7fc70d240640 10 monclient: get_monmap_and_config 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.084+0000 7fc70d240640 10 monclient: build_initial_monmap 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.084+0000 7fc70d240640 10 monclient: monmap: 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:07.084948+0000 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:07.084948+0000 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:07.084 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 1 Processor -- start 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 1 -- start start 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient: init 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-a con 0x7fc7080705b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-c con 0x7fc7081402d0 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.085+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-b con 0x7fc708149ae0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:07.086 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.086+0000 7fc70d240640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080718c0 con 0x7fc7080705b0 2024-03-25T19:44:07.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.086+0000 7fc70d240640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080714c0 con 0x7fc7081402d0 2024-03-25T19:44:07.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.086+0000 7fc70d240640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708072b50 con 0x7fc708149ae0 2024-03-25T19:44:07.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.086+0000 7fc70d240640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:07.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.086+0000 7fc70d240640 10 monclient(hunting): authenticate will time out at 6723.838941s 2024-03-25T19:44:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.093+0000 7fc706d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc7080705b0 0x7fc70813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52512/0 (socket says 172.21.15.29:52512) 2024-03-25T19:44:07.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.093+0000 7fc706d9d640 1 -- 172.21.15.29:0/4245449588 learned_addr learned my addr 172.21.15.29:0/4245449588 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754068128 0 0) 0x7fc7080718c0 con 0x7fc7080705b0 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:44:07.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.112+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0003520 con 0x7fc7080705b0 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469808712 0 0) 0x7fc7080714c0 con 0x7fc7081402d0 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:44:07.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.115+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7080718c0 con 0x7fc7081402d0 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520743239 0 0) 0x7fc6f0003520 con 0x7fc7080705b0 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 >> v1:172.21.15.29:6790/0 conn(0x7fc708149ae0 legacy=0x7fc70814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 >> v1:172.21.15.196:6789/0 conn(0x7fc7081402d0 legacy=0x7fc708140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 monclient: found mon.noname-a 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70813dd10 con 0x7fc7080705b0 2024-03-25T19:44:07.126 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc6f8001690 con 0x7fc7080705b0 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc705d9b640 10 monclient: dump: 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:07.127 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:07.128 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc70d240640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:44:07.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.126+0000 7fc70d240640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f80017f0 con 0x7fc7080705b0 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 1 -- 172.21.15.29:0/4245449588 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc6f8001c00 con 0x7fc7080705b0 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc70d240640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc70d240640 4 set_mon_vals no callback set 2024-03-25T19:44:07.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.142+0000 7fc705d9b640 10 monclient: dump: 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:07.143 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.143+0000 7fc70d240640 10 monclient: shutdown 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.143+0000 7fc70d240640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.143+0000 7fc70d240640 1 -- 172.21.15.29:0/4245449588 >> v1:172.21.15.29:6789/0 conn(0x7fc7080705b0 legacy=0x7fc70813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.143+0000 7fc70d240640 1 -- 172.21.15.29:0/4245449588 shutdown_connections 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.143+0000 7fc70d240640 1 -- 172.21.15.29:0/4245449588 >> 172.21.15.29:0/4245449588 conn(0x7fc7080b5d10 msgr2=0x7fc7080b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.144+0000 7fc70d240640 1 -- 172.21.15.29:0/4245449588 shutdown_connections 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.144+0000 7fc70d240640 1 -- 172.21.15.29:0/4245449588 wait complete. 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 10 monclient: build_initial_monmap 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 10 monclient: monmap: 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:07.145 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 librados: starting msgr at 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 librados: starting objecter 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 Processor -- start 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 -- start start 2024-03-25T19:44:07.146 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 librados: setting wanted keys 2024-03-25T19:44:07.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 1 librados: calling monclient init 2024-03-25T19:44:07.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.145+0000 7fc70d240640 10 monclient: init 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-a con 0x7fc7080705b0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-c con 0x7fc7081402d0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient(hunting): picked mon.noname-b con 0x7fc708149ae0 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708144230 con 0x7fc7080705b0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70813ec00 con 0x7fc708149ae0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7081444a0 con 0x7fc7081402d0 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.146+0000 7fc70d240640 10 monclient(hunting): authenticate will time out at 6723.899462s 2024-03-25T19:44:07.148 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.147+0000 7fc70659c640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fc7081402d0 0x7fc708133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:36220/0 (socket says 172.21.15.29:36220) 2024-03-25T19:44:07.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.147+0000 7fc70659c640 1 -- 172.21.15.29:0/3155579558 learned_addr learned my addr 172.21.15.29:0/3155579558 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:07.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.149+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc7081402d0 2024-03-25T19:44:07.149 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.149+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.155+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc708149ae0 2024-03-25T19:44:07.154 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.155+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724934790 0 0) 0x7fc7081444a0 con 0x7fc7081402d0 2024-03-25T19:44:07.158 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:44:07.159 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.159+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f4004eb0 con 0x7fc7081402d0 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572699328 0 0) 0x7fc70813ec00 con 0x7fc708149ae0 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:44:07.163 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.164+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7081444a0 con 0x7fc708149ae0 2024-03-25T19:44:07.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2144084710 0 0) 0x7fc7081444a0 con 0x7fc708149ae0 2024-03-25T19:44:07.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc70813ec00 con 0x7fc708149ae0 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc700001b00 con 0x7fc708149ae0 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.174+0000 7fc6ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:07.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.180+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc7080705b0 2024-03-25T19:44:07.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.180+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329548667 0 0) 0x7fc6f4004eb0 con 0x7fc7081402d0 2024-03-25T19:44:07.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7081444a0 con 0x7fc7081402d0 2024-03-25T19:44:07.182 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc6fc001a10 con 0x7fc7081402d0 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.183+0000 7fc6ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:07.183 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3095066900 0 0) 0x7fc70813ec00 con 0x7fc708149ae0 2024-03-25T19:44:07.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 >> v1:172.21.15.29:6790/0 conn(0x7fc7081402d0 legacy=0x7fc708133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 >> v1:172.21.15.29:6789/0 conn(0x7fc7080705b0 legacy=0x7fc70813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:07.191 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 1 monclient: found mon.b 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7081429c0 con 0x7fc708149ae0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc6ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 10 monclient: _renew_subs 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7080af750 con 0x7fc708149ae0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 10 monclient: _renew_subs 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:07.192 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc708142500 con 0x7fc708149ae0 2024-03-25T19:44:07.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.191+0000 7fc70d240640 1 librados: init done 2024-03-25T19:44:07.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.192+0000 7fc70d240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.192+0000 7fc70d240640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.192+0000 7fc70d240640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:07.193 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.192+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc708142c10 con 0x7fc708149ae0 2024-03-25T19:44:07.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.461+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc700001ff0 con 0x7fc708149ae0 2024-03-25T19:44:07.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc6ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fc700003b10 con 0x7fc708149ae0 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc6ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc6ef7fe640 10 monclient: dump: 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:07.462 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:07.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.462+0000 7fc70559a640 4 set_mon_vals no callback set 2024-03-25T19:44:07.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.463+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fc700003cf0 con 0x7fc708149ae0 2024-03-25T19:44:07.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.463+0000 7fc706d9d640 10 monclient: get_auth_request con 0x7fc6f406ddf0 auth_method 0 2024-03-25T19:44:07.463 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.463+0000 7fc6ef7fe640 10 client.4690.objecter ms_handle_connect 0x7fc6f406ddf0 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fc700053820 con 0x7fc708149ae0 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 10 client.4690.objecter ms_dispatch 0x7fc708000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc70004f160 con 0x7fc708149ae0 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.468+0000 7fc6ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:07.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.747+0000 7fc70d240640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.747+0000 7fc70d240640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.747+0000 7fc70d240640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:07.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.747+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc708143110 con 0x7fc708149ae0 2024-03-25T19:44:07.817 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.818+0000 7fc6ef7fe640 1 -- 172.21.15.29:0/3155579558 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fc700051260 con 0x7fc708149ae0 2024-03-25T19:44:07.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.818+0000 7fc6ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.818 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.818+0000 7fc6ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:07.818 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:07.818 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:07.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 10 librados: watch_flush enter 2024-03-25T19:44:07.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 10 librados: watch_flush exit 2024-03-25T19:44:07.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T19:44:07.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 10 client.4690.objecter successfully canceled tick 2024-03-25T19:44:07.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 >> v1:172.21.15.29:6800/2314944989 conn(0x7fc6f406ddf0 legacy=0x7fc6f40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 10 monclient: shutdown 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 >> v1:172.21.15.196:6789/0 conn(0x7fc708149ae0 legacy=0x7fc7081e1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 shutdown_connections 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 >> 172.21.15.29:0/3155579558 conn(0x7fc7080b5d10 msgr2=0x7fc7080b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 shutdown_connections 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 -- 172.21.15.29:0/3155579558 wait complete. 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:07.820+0000 7fc70d240640 1 librados: shutdown 2024-03-25T19:44:07.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:11.522 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:41.686 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.536 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:41.700 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.561 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:41.725 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.599 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:41.762 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.607 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:41.769 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.609 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:41.772 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.619 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:41.783 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.638 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:10:41.801 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:11.668 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:41.831 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.639 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:14.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.757+0000 7fb0408fb640 10 monclient: get_monmap_and_config 2024-03-25T19:44:14.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.757+0000 7fb0408fb640 10 monclient: build_initial_monmap 2024-03-25T19:44:14.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.757+0000 7fb0408fb640 10 monclient: monmap: 2024-03-25T19:44:14.757 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:14.758250+0000 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:14.758250+0000 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 1 Processor -- start 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 1 -- start start 2024-03-25T19:44:14.758 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient: init 2024-03-25T19:44:14.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:14.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:14.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-b con 0x7fb03c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:14.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-a con 0x7fb03c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:14.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-c con 0x7fb03c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:14.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c142520 con 0x7fb03c13f710 2024-03-25T19:44:14.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0b28a0 con 0x7fb03c1430c0 2024-03-25T19:44:14.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0b2a30 con 0x7fb03c0a98f0 2024-03-25T19:44:14.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:14.760 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.758+0000 7fb0408fb640 10 monclient(hunting): authenticate will time out at 6731.511788s 2024-03-25T19:44:14.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.768+0000 7fb031d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb03c13f710 0x7fb03c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56980/0 (socket says 172.21.15.29:56980) 2024-03-25T19:44:14.767 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.768+0000 7fb031d9b640 1 -- 172.21.15.29:0/4049782286 learned_addr learned my addr 172.21.15.29:0/4049782286 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695867978 0 0) 0x7fb03c142520 con 0x7fb03c13f710 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:44:14.787 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.787+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb028003520 con 0x7fb03c13f710 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718541242 0 0) 0x7fb03c0b28a0 con 0x7fb03c1430c0 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c142520 con 0x7fb03c1430c0 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865169355 0 0) 0x7fb03c0b2a30 con 0x7fb03c0a98f0 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.788 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:44:14.789 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.788+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c0b28a0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560169216 0 0) 0x7fb03c142520 con 0x7fb03c1430c0 2024-03-25T19:44:14.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 >> v1:172.21.15.29:6790/0 conn(0x7fb03c0a98f0 legacy=0x7fb03c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.808 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 >> v1:172.21.15.29:6789/0 conn(0x7fb03c13f710 legacy=0x7fb03c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 monclient: found mon.noname-c 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0b44c0 con 0x7fb03c1430c0 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb0340019f0 con 0x7fb03c1430c0 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.808+0000 7fb039d9b640 10 monclient: dump: 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:14.809 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.809+0000 7fb0408fb640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:44:14.810 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.809+0000 7fb0408fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb034001b50 con 0x7fb03c1430c0 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 1 -- 172.21.15.29:0/4049782286 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb034001f60 con 0x7fb03c1430c0 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb0408fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb0408fb640 4 set_mon_vals no callback set 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.835+0000 7fb039d9b640 10 monclient: dump: 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:14.835 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:14.836 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 10 monclient: shutdown 2024-03-25T19:44:14.837 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 1 -- 172.21.15.29:0/4049782286 >> v1:172.21.15.196:6789/0 conn(0x7fb03c1430c0 legacy=0x7fb03c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 1 -- 172.21.15.29:0/4049782286 shutdown_connections 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 1 -- 172.21.15.29:0/4049782286 >> 172.21.15.29:0/4049782286 conn(0x7fb03c0b7ea0 msgr2=0x7fb03c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 1 -- 172.21.15.29:0/4049782286 shutdown_connections 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.836+0000 7fb0408fb640 1 -- 172.21.15.29:0/4049782286 wait complete. 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.837+0000 7fb0408fb640 10 monclient: build_initial_monmap 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.837+0000 7fb0408fb640 10 monclient: monmap: 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:14.838 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 librados: starting msgr at 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 librados: starting objecter 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 Processor -- start 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 -- start start 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 librados: setting wanted keys 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 1 librados: calling monclient init 2024-03-25T19:44:14.839 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.838+0000 7fb0408fb640 10 monclient: init 2024-03-25T19:44:14.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:14.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:14.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-c con 0x7fb03c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:14.840 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-a con 0x7fb03c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient(hunting): picked mon.noname-b con 0x7fb03c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c13c190 con 0x7fb03c13f710 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0b7cf0 con 0x7fb03c1430c0 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c13ad50 con 0x7fb03c0a98f0 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:14.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.839+0000 7fb0408fb640 10 monclient(hunting): authenticate will time out at 6731.592114s 2024-03-25T19:44:14.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.844+0000 7fb031d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb03c13f710 0x7fb03c1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57042/0 (socket says 172.21.15.29:57042) 2024-03-25T19:44:14.844 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.844+0000 7fb031d9b640 1 -- 172.21.15.29:0/1548571198 learned_addr learned my addr 172.21.15.29:0/1548571198 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:14.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.845+0000 7fb0337fe640 10 client.?.objecter ms_handle_connect 0x7fb03c13f710 2024-03-25T19:44:14.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.845+0000 7fb0337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.846+0000 7fb0337fe640 10 client.?.objecter ms_handle_connect 0x7fb03c0a98f0 2024-03-25T19:44:14.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.846+0000 7fb0337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785849687 0 0) 0x7fb03c13ad50 con 0x7fb03c0a98f0 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:44:14.853 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.853+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c004eb0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.854 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646983628 0 0) 0x7fb028003520 con 0x7fb03c13f710 2024-03-25T19:44:14.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:44:14.855 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.855+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c13ad50 con 0x7fb03c13f710 2024-03-25T19:44:14.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.861+0000 7fb0337fe640 10 client.?.objecter ms_handle_connect 0x7fb03c1430c0 2024-03-25T19:44:14.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.861+0000 7fb0337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357816590 0 0) 0x7fb01c004eb0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.872 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb028003520 con 0x7fb03c0a98f0 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb02c0018f0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 10 monclient(hunting): dump: 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.873 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.874 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.872+0000 7fb0337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:14.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67279472 0 0) 0x7fb03c13ad50 con 0x7fb03c13f710 2024-03-25T19:44:14.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c004eb0 con 0x7fb03c13f710 2024-03-25T19:44:14.875 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb0240017b0 con 0x7fb03c13f710 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.874+0000 7fb0337fe640 10 monclient(hunting): dump: 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.876 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:14.877 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:14.877 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:14.877 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:14.877 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:14.882 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041434192 0 0) 0x7fb03c0b7cf0 con 0x7fb03c1430c0 2024-03-25T19:44:14.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:44:14.883 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.883+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c13ad50 con 0x7fb03c1430c0 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 302085121 0 0) 0x7fb028003520 con 0x7fb03c0a98f0 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 >> v1:172.21.15.196:6789/0 conn(0x7fb03c1430c0 legacy=0x7fb03c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 >> v1:172.21.15.29:6789/0 conn(0x7fb03c13f710 legacy=0x7fb03c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 1 monclient: found mon.c 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c13afc0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.886 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 5 monclient: authenticate success, global_id 4688 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 10 monclient: _renew_subs 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c13b2a0 con 0x7fb03c0a98f0 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 10 monclient: _renew_subs 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c12fa90 con 0x7fb03c0a98f0 2024-03-25T19:44:14.887 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.886+0000 7fb0408fb640 1 librados: init done 2024-03-25T19:44:14.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.887+0000 7fb0408fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:14.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.887+0000 7fb0408fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:14.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.887+0000 7fb0408fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:14.888 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:14.887+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c0b3ef0 con 0x7fb03c0a98f0 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb02c003610 con 0x7fb03c0a98f0 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fb02c003920 con 0x7fb03c0a98f0 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb0337fe640 10 monclient: dump: 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.207 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:15.208 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:15.208 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:15.208 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:15.208 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:15.208 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.207+0000 7fb03959a640 4 set_mon_vals no callback set 2024-03-25T19:44:15.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.208+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fb02c019930 con 0x7fb03c0a98f0 2024-03-25T19:44:15.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.209+0000 7fb031d9b640 10 monclient: get_auth_request con 0x7fb01c06df50 auth_method 0 2024-03-25T19:44:15.209 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.209+0000 7fb0337fe640 10 client.4688.objecter ms_handle_connect 0x7fb01c06df50 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fb02c0535f0 con 0x7fb03c0a98f0 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 10 client.4688.objecter ms_dispatch 0x7fb03c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb02c04ef30 con 0x7fb03c0a98f0 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:15.215 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.215+0000 7fb0337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:15.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.440+0000 7fb0408fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.440+0000 7fb0408fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.440+0000 7fb0408fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:15.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.440+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb03c0627b0 con 0x7fb03c0a98f0 2024-03-25T19:44:15.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.475+0000 7fb0337fe640 1 -- 172.21.15.29:0/1548571198 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fb02c051030 con 0x7fb03c0a98f0 2024-03-25T19:44:15.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.475+0000 7fb0337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.475+0000 7fb0337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:15.476 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:15.476 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:15.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 10 librados: watch_flush enter 2024-03-25T19:44:15.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 10 librados: watch_flush exit 2024-03-25T19:44:15.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-25T19:44:15.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 10 client.4688.objecter successfully canceled tick 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 >> v1:172.21.15.29:6800/2314944989 conn(0x7fb01c06df50 legacy=0x7fb01c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 10 monclient: shutdown 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 >> v1:172.21.15.29:6790/0 conn(0x7fb03c0a98f0 legacy=0x7fb03c13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 shutdown_connections 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 >> 172.21.15.29:0/1548571198 conn(0x7fb03c0b7ea0 msgr2=0x7fb03c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 shutdown_connections 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 -- 172.21.15.29:0/1548571198 wait complete. 2024-03-25T19:44:15.479 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:15.478+0000 7fb0408fb640 1 librados: shutdown 2024-03-25T19:44:15.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:22.296 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:22.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.412+0000 7f5581e51640 10 monclient: get_monmap_and_config 2024-03-25T19:44:22.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.412+0000 7f5581e51640 10 monclient: build_initial_monmap 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.412+0000 7f5581e51640 10 monclient: monmap: 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:22.413947+0000 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:22.413947+0000 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:22.413 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.413+0000 7f5581e51640 1 Processor -- start 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.413+0000 7f5581e51640 1 -- start start 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.413+0000 7f5581e51640 10 monclient: init 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.413+0000 7f5581e51640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.413+0000 7f5581e51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-a con 0x7f557c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-b con 0x7f557c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-c con 0x7f557c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0af940 con 0x7f557c0b4d60 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0b0a70 con 0x7f557c14be90 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0b0d50 con 0x7f557c0b1730 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:22.415 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.414+0000 7f5581e51640 10 monclient(hunting): authenticate will time out at 6739.168214s 2024-03-25T19:44:22.417 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.417+0000 7f557b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f557c0b4d60 0x7f557c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51086/0 (socket says 172.21.15.29:51086) 2024-03-25T19:44:22.418 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.417+0000 7f557b7fe640 1 -- 172.21.15.29:0/2680852845 learned_addr learned my addr 172.21.15.29:0/2680852845 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466149846 0 0) 0x7f557c0b0a70 con 0x7f557c14be90 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:44:22.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.427+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5564003520 con 0x7f557c14be90 2024-03-25T19:44:22.440 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677709320 0 0) 0x7f5564003520 con 0x7f557c14be90 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 >> v1:172.21.15.29:6790/0 conn(0x7f557c0b1730 legacy=0x7f557c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 >> v1:172.21.15.29:6789/0 conn(0x7f557c0b4d60 legacy=0x7f557c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c0b1030 con 0x7f557c14be90 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5574001a10 con 0x7f557c14be90 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:22.441 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f557a7fc640 10 monclient: dump: 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f5581e51640 5 monclient: authenticate success, global_id 4699 2024-03-25T19:44:22.442 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.440+0000 7f5581e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:22.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f557a7fc640 1 -- 172.21.15.29:0/2680852845 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5574001b70 con 0x7f557c14be90 2024-03-25T19:44:22.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f557a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:22.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:22.443 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 4 set_mon_vals no callback set 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 10 monclient: shutdown 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 1 -- 172.21.15.29:0/2680852845 >> v1:172.21.15.196:6789/0 conn(0x7f557c14be90 legacy=0x7f557c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 1 -- 172.21.15.29:0/2680852845 shutdown_connections 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.443+0000 7f5581e51640 1 -- 172.21.15.29:0/2680852845 >> 172.21.15.29:0/2680852845 conn(0x7f557c0617a0 msgr2=0x7f557c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:22.444 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.444+0000 7f5581e51640 1 -- 172.21.15.29:0/2680852845 shutdown_connections 2024-03-25T19:44:22.445 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.444+0000 7f5581e51640 1 -- 172.21.15.29:0/2680852845 wait complete. 2024-03-25T19:44:22.445 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.444+0000 7f5581e51640 10 monclient: build_initial_monmap 2024-03-25T19:44:22.445 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.444+0000 7f5581e51640 10 monclient: monmap: 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 librados: starting msgr at 2024-03-25T19:44:22.446 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 librados: starting objecter 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 Processor -- start 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 -- start start 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 librados: setting wanted keys 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 1 librados: calling monclient init 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.445+0000 7f5581e51640 10 monclient: init 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:22.447 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-a con 0x7f557c0b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-c con 0x7f557c0b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient(hunting): picked mon.noname-b con 0x7f557c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c140410 con 0x7f557c0b1730 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c13bf20 con 0x7f557c14be90 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c1377c0 con 0x7f557c0b4d60 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:22.448 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.446+0000 7f5581e51640 10 monclient(hunting): authenticate will time out at 6739.200146s 2024-03-25T19:44:22.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.454+0000 7f557affd640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f557c0b4d60 0x7f557c136cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:45056/0 (socket says 172.21.15.29:45056) 2024-03-25T19:44:22.455 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.454+0000 7f557affd640 1 -- 172.21.15.29:0/793745701 learned_addr learned my addr 172.21.15.29:0/793745701 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.456+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f557c0b4d60 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.456+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.464+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f557c14be90 2024-03-25T19:44:22.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.464+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.466+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f557c0b1730 2024-03-25T19:44:22.466 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.466+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172985054 0 0) 0x7f557c1377c0 con 0x7f557c0b4d60 2024-03-25T19:44:22.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.474 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:44:22.475 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.474+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568004eb0 con 0x7f557c0b4d60 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754771428 0 0) 0x7f557c13bf20 con 0x7f557c14be90 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c1377c0 con 0x7f557c14be90 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790531230 0 0) 0x7f557c140410 con 0x7f557c0b1730 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.477 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:44:22.478 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.476+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c13bf20 con 0x7f557c0b1730 2024-03-25T19:44:22.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2472484609 0 0) 0x7f5568004eb0 con 0x7f557c0b4d60 2024-03-25T19:44:22.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f557c140410 con 0x7f557c0b4d60 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5570001a30 con 0x7f557c0b4d60 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 10 monclient(hunting): dump: 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:22.486 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:22.487 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.485+0000 7f5563fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2937976386 0 0) 0x7f557c1377c0 con 0x7f557c14be90 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5568004eb0 con 0x7f557c14be90 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f5574001e80 con 0x7f557c14be90 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.488+0000 7f5563fff640 10 monclient(hunting): dump: 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.489 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.490 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:22.490 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:22.490 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:22.490 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300301035 0 0) 0x7f557c13bf20 con 0x7f557c0b1730 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f557c1377c0 con 0x7f557c0b1730 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f556c001860 con 0x7f557c0b1730 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.494+0000 7f5563fff640 10 monclient(hunting): dump: 2024-03-25T19:44:22.495 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:22.496 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2775183390 0 0) 0x7f557c140410 con 0x7f557c0b4d60 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 >> v1:172.21.15.196:6789/0 conn(0x7f557c14be90 legacy=0x7f557c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 >> v1:172.21.15.29:6789/0 conn(0x7f557c0b1730 legacy=0x7f557c1365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 1 monclient: found mon.c 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c137950 con 0x7f557c0b4d60 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:22.505 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 5 monclient: authenticate success, global_id 4694 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 10 monclient: _renew_subs 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557c138140 con 0x7f557c0b4d60 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 10 monclient: _renew_subs 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f557c144800 con 0x7f557c0b4d60 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.503+0000 7f5581e51640 1 librados: init done 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.504+0000 7f5581e51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.504+0000 7f5581e51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.504+0000 7f5581e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:22.506 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.504+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557c141990 con 0x7f557c0b4d60 2024-03-25T19:44:22.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5570001c70 con 0x7f557c0b4d60 2024-03-25T19:44:22.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:22.825 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f55700038c0 con 0x7f557c0b4d60 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f5563fff640 10 monclient: dump: 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:22.826 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:22.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.824+0000 7f55797fa640 4 set_mon_vals no callback set 2024-03-25T19:44:22.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.825+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f5570003ae0 con 0x7f557c0b4d60 2024-03-25T19:44:22.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.826+0000 7f557b7fe640 10 monclient: get_auth_request con 0x7f556806e0f0 auth_method 0 2024-03-25T19:44:22.827 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.826+0000 7f5563fff640 10 client.4694.objecter ms_handle_connect 0x7f556806e0f0 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f5570054680 con 0x7f557c0b4d60 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 10 client.4694.objecter ms_dispatch 0x7f557c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5570050190 con 0x7f557c0b4d60 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:22.832+0000 7f5563fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:23.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.068+0000 7f5581e51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.068+0000 7f5581e51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.068+0000 7f5581e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:23.069 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.068+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f557c141550 con 0x7f557c0b4d60 2024-03-25T19:44:23.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.117+0000 7f5563fff640 1 -- 172.21.15.29:0/793745701 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f55700520c0 con 0x7f557c0b4d60 2024-03-25T19:44:23.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.117+0000 7f5563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.117+0000 7f5563fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:23.118 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:23.118 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.119+0000 7f5581e51640 10 librados: watch_flush enter 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.119+0000 7f5581e51640 10 librados: watch_flush exit 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.119+0000 7f5581e51640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.119+0000 7f5581e51640 10 client.4694.objecter successfully canceled tick 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 >> v1:172.21.15.29:6800/2314944989 conn(0x7f556806e0f0 legacy=0x7f55680705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 10 monclient: shutdown 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 >> v1:172.21.15.29:6790/0 conn(0x7f557c0b4d60 legacy=0x7f557c136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 shutdown_connections 2024-03-25T19:44:23.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 >> 172.21.15.29:0/793745701 conn(0x7f557c0617a0 msgr2=0x7f557c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:23.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 shutdown_connections 2024-03-25T19:44:23.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 -- 172.21.15.29:0/793745701 wait complete. 2024-03-25T19:44:23.122 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:23.120+0000 7f5581e51640 1 librados: shutdown 2024-03-25T19:44:23.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:29.939 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:30.056 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.056+0000 7fbd91e27640 10 monclient: get_monmap_and_config 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.056+0000 7fbd91e27640 10 monclient: build_initial_monmap 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.056+0000 7fbd91e27640 10 monclient: monmap: 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:30.057991+0000 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:30.057991+0000 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:30.057 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.058 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.057+0000 7fbd91e27640 1 Processor -- start 2024-03-25T19:44:30.058 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.057+0000 7fbd91e27640 1 -- start start 2024-03-25T19:44:30.058 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.057+0000 7fbd91e27640 10 monclient: init 2024-03-25T19:44:30.059 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:30.059 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:30.059 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-b con 0x7fbd8c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-c con 0x7fbd8c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-a con 0x7fbd8c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c142520 con 0x7fbd8c1430c0 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c0b28a0 con 0x7fbd8c13f710 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c0b2a30 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:30.060 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.058+0000 7fbd91e27640 10 monclient(hunting): authenticate will time out at 6746.812053s 2024-03-25T19:44:30.063 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.062+0000 7fbd8b7fe640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7fbd8c0a98f0 0x7fbd8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:50592/0 (socket says 172.21.15.29:50592) 2024-03-25T19:44:30.063 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.062+0000 7fbd8b7fe640 1 -- 172.21.15.29:0/1929843691 learned_addr learned my addr 172.21.15.29:0/1929843691 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:30.075 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163421280 0 0) 0x7fbd8c0b2a30 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:44:30.076 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.075+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74003520 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604915534 0 0) 0x7fbd74003520 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 >> v1:172.21.15.196:6789/0 conn(0x7fbd8c13f710 legacy=0x7fbd8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 >> v1:172.21.15.29:6789/0 conn(0x7fbd8c1430c0 legacy=0x7fbd8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.29:6790/0 2024-03-25T19:44:30.092 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c0b44c0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbd7c001690 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd8a7fc640 10 monclient: dump: 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:30.093 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd91e27640 5 monclient: authenticate success, global_id 4697 2024-03-25T19:44:30.094 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.091+0000 7fbd91e27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 <== mon.2 v1:172.21.15.29:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd7c0017f0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 1 -- 172.21.15.29:0/1929843691 <== mon.2 v1:172.21.15.29:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbd7c001c00 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd91e27640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd91e27640 4 set_mon_vals no callback set 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.108+0000 7fbd8a7fc640 10 monclient: dump: 2024-03-25T19:44:30.109 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:30.110 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 10 monclient: shutdown 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 1 -- 172.21.15.29:0/1929843691 >> v1:172.21.15.29:6790/0 conn(0x7fbd8c0a98f0 legacy=0x7fbd8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 1 -- 172.21.15.29:0/1929843691 shutdown_connections 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 1 -- 172.21.15.29:0/1929843691 >> 172.21.15.29:0/1929843691 conn(0x7fbd8c0b7ea0 msgr2=0x7fbd8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 1 -- 172.21.15.29:0/1929843691 shutdown_connections 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.109+0000 7fbd91e27640 1 -- 172.21.15.29:0/1929843691 wait complete. 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.110+0000 7fbd91e27640 10 monclient: build_initial_monmap 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.110+0000 7fbd91e27640 10 monclient: monmap: 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:30.112 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 librados: starting msgr at 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 librados: starting objecter 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 Processor -- start 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 -- start start 2024-03-25T19:44:30.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 librados: setting wanted keys 2024-03-25T19:44:30.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 1 librados: calling monclient init 2024-03-25T19:44:30.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.111+0000 7fbd91e27640 10 monclient: init 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-a con 0x7fbd8c0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-b con 0x7fbd8c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient(hunting): picked mon.noname-c con 0x7fbd8c1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c139a40 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c133f20 con 0x7fbd8c13f710 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c0ab950 con 0x7fbd8c1430c0 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:30.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.112+0000 7fbd91e27640 10 monclient(hunting): authenticate will time out at 6746.866117s 2024-03-25T19:44:30.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.119+0000 7fbd8b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbd8c0a98f0 0x7fbd8c138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41956/0 (socket says 172.21.15.29:41956) 2024-03-25T19:44:30.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.119+0000 7fbd8b7fe640 1 -- 172.21.15.29:0/1315223369 learned_addr learned my addr 172.21.15.29:0/1315223369 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:30.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.120+0000 7fbd73fff640 10 client.?.objecter ms_handle_connect 0x7fbd8c0a98f0 2024-03-25T19:44:30.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.120+0000 7fbd73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:30.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.123+0000 7fbd73fff640 10 client.?.objecter ms_handle_connect 0x7fbd8c13f710 2024-03-25T19:44:30.123 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.123+0000 7fbd73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:30.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.123+0000 7fbd73fff640 10 client.?.objecter ms_handle_connect 0x7fbd8c1430c0 2024-03-25T19:44:30.124 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.123+0000 7fbd73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634374148 0 0) 0x7fbd8c0ab950 con 0x7fbd8c1430c0 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:44:30.132 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.131+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd78004eb0 con 0x7fbd8c1430c0 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197124904 0 0) 0x7fbd8c139a40 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:44:30.133 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.132+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd8c0ab950 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.134 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162323018 0 0) 0x7fbd8c133f20 con 0x7fbd8c13f710 2024-03-25T19:44:30.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:44:30.135 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.134+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd8c139a40 con 0x7fbd8c13f710 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781147306 0 0) 0x7fbd8c0ab950 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd8c133f20 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbd7c001b40 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 10 monclient(hunting): dump: 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:30.143 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:30.144 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:30.144 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.144 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.142+0000 7fbd73fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3523535991 0 0) 0x7fbd8c139a40 con 0x7fbd8c13f710 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd8c0ab950 con 0x7fbd8c13f710 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbd80001830 con 0x7fbd8c13f710 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.161+0000 7fbd73fff640 10 monclient(hunting): dump: 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.162 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:30.163 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068293705 0 0) 0x7fbd8c133f20 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 >> v1:172.21.15.29:6790/0 conn(0x7fbd8c1430c0 legacy=0x7fbd8c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 >> v1:172.21.15.196:6789/0 conn(0x7fbd8c13f710 legacy=0x7fbd8c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 1 monclient: found mon.a 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c0abc30 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 10 monclient: _renew_subs 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8c0abf10 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 10 monclient: _renew_subs 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:30.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8c139370 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.162+0000 7fbd91e27640 1 librados: init done 2024-03-25T19:44:30.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.163+0000 7fbd91e27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:30.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.163+0000 7fbd91e27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:30.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.163+0000 7fbd91e27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:30.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.163+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8c0627b0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.380 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd7c004a90 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbd7c004da0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd73fff640 10 monclient: dump: 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:30.381 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:30.382 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:30.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.380+0000 7fbd89ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:30.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.487+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbd7c01ad70 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.488+0000 7fbd8affd640 10 monclient: get_auth_request con 0x7fbd7806df80 auth_method 0 2024-03-25T19:44:30.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.488+0000 7fbd73fff640 10 client.4707.objecter ms_handle_connect 0x7fbd7806df80 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fbd7c0548a0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 10 client.4707.objecter ms_dispatch 0x7fbd8c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd7c0503b0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.496 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:30.497 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.495+0000 7fbd73fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:30.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.746+0000 7fbd91e27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.746+0000 7fbd91e27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.746+0000 7fbd91e27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.29:6789/0 2024-03-25T19:44:30.747 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.746+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 --> v1:172.21.15.29:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd8c0b7180 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.767+0000 7fbd73fff640 1 -- 172.21.15.29:0/1315223369 <== mon.0 v1:172.21.15.29:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fbd7c0522e0 con 0x7fbd8c0a98f0 2024-03-25T19:44:30.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.767+0000 7fbd73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.768 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.767+0000 7fbd73fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:30.768 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:30.769 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 10 librados: watch_flush enter 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 10 librados: watch_flush exit 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbd7806df80 legacy=0x7fbd78070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 10 monclient: shutdown 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:30.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 >> v1:172.21.15.29:6789/0 conn(0x7fbd8c0a98f0 legacy=0x7fbd8c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.770+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 shutdown_connections 2024-03-25T19:44:30.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.771+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 >> 172.21.15.29:0/1315223369 conn(0x7fbd8c0b7ea0 msgr2=0x7fbd8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:30.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.771+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 shutdown_connections 2024-03-25T19:44:30.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.771+0000 7fbd91e27640 1 -- 172.21.15.29:0/1315223369 wait complete. 2024-03-25T19:44:30.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:30.771+0000 7fbd91e27640 1 librados: shutdown 2024-03-25T19:44:30.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:31.625 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:01.788 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.629 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:01.792 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.670 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:01.833 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.673 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:01.836 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.685 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:01.849 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.686 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:01.848 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.691 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:01.855 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.709 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:01.871 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:31.717 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:01.880 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.590 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.705+0000 7f7a05ad3640 10 monclient: get_monmap_and_config 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.705+0000 7f7a05ad3640 10 monclient: build_initial_monmap 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.705+0000 7f7a05ad3640 10 monclient: monmap: 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.705 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:37.706594+0000 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:37.706594+0000 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:37.706 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.706+0000 7f7a05ad3640 1 Processor -- start 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.706+0000 7f7a05ad3640 1 -- start start 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.706+0000 7f7a05ad3640 10 monclient: init 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-c con 0x7f7a000b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-a con 0x7f7a000b1730 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:37.707 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-b con 0x7f7a0014be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:37.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000af940 con 0x7f7a000b1730 2024-03-25T19:44:37.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000b0a70 con 0x7f7a000b4d60 2024-03-25T19:44:37.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000b0d50 con 0x7f7a0014be90 2024-03-25T19:44:37.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:37.708 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.707+0000 7f7a05ad3640 10 monclient(hunting): authenticate will time out at 6754.460526s 2024-03-25T19:44:37.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.714+0000 7f79ff7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7a000b4d60 0x7f7a0013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:57260/0 (socket says 172.21.15.29:57260) 2024-03-25T19:44:37.713 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.714+0000 7f79ff7fe640 1 -- 172.21.15.29:0/305143892 learned_addr learned my addr 172.21.15.29:0/305143892 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184085736 0 0) 0x7f7a000af940 con 0x7f7a000b1730 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:44:37.731 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.732+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e8003520 con 0x7f7a000b1730 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643083939 0 0) 0x7f7a000b0a70 con 0x7f7a000b4d60 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:44:37.737 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.737+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a000af940 con 0x7f7a000b4d60 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418785654 0 0) 0x7f7a000b0d50 con 0x7f7a0014be90 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:44:37.738 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.738+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a000b0a70 con 0x7f7a0014be90 2024-03-25T19:44:37.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4147552404 0 0) 0x7f79e8003520 con 0x7f7a000b1730 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 >> v1:172.21.15.29:6790/0 conn(0x7f7a0014be90 legacy=0x7f7a0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 >> v1:172.21.15.196:6789/0 conn(0x7f7a000b4d60 legacy=0x7f7a0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 monclient: found mon.noname-a 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000b1030 con 0x7f7a000b1730 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f4001690 con 0x7f7a000b1730 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f7a05ad3640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f7a05ad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.754+0000 7f79fe7fc640 10 monclient: dump: 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.755+0000 7f7a05ad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:37.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79f40017f0 con 0x7f7a000b1730 2024-03-25T19:44:37.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:37.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 1 -- 172.21.15.29:0/305143892 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f4001c00 con 0x7f7a000b1730 2024-03-25T19:44:37.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.772 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f79fe7fc640 10 monclient: dump: 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f7a05ad3640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:37.773 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.772+0000 7f7a05ad3640 4 set_mon_vals no callback set 2024-03-25T19:44:37.774 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 10 monclient: shutdown 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 1 -- 172.21.15.29:0/305143892 >> v1:172.21.15.29:6789/0 conn(0x7f7a000b1730 legacy=0x7f7a000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 1 -- 172.21.15.29:0/305143892 shutdown_connections 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 1 -- 172.21.15.29:0/305143892 >> 172.21.15.29:0/305143892 conn(0x7f7a000617a0 msgr2=0x7f7a00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 1 -- 172.21.15.29:0/305143892 shutdown_connections 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.773+0000 7f7a05ad3640 1 -- 172.21.15.29:0/305143892 wait complete. 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.774+0000 7f7a05ad3640 10 monclient: build_initial_monmap 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.774+0000 7f7a05ad3640 10 monclient: monmap: 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:37.775 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.774+0000 7f7a05ad3640 1 librados: starting msgr at 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.774+0000 7f7a05ad3640 1 librados: starting objecter 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 Processor -- start 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 -- start start 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 librados: setting wanted keys 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 librados: calling monclient init 2024-03-25T19:44:37.776 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient: init 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-c con 0x7f7a000b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-b con 0x7f7a000b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0014be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a001d7860 con 0x7f7a0014be90 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a00140d00 con 0x7f7a000b4d60 2024-03-25T19:44:37.777 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a00141a30 con 0x7f7a000b1730 2024-03-25T19:44:37.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.775+0000 7f7a05ad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:37.778 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.776+0000 7f7a05ad3640 10 monclient(hunting): authenticate will time out at 6754.528871s 2024-03-25T19:44:37.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.798+0000 7f79ff7fe640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f7a000b1730 0x7f7a001405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:52176/0 (socket says 172.21.15.29:52176) 2024-03-25T19:44:37.797 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.798+0000 7f79ff7fe640 1 -- 172.21.15.29:0/637533120 learned_addr learned my addr 172.21.15.29:0/637533120 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:37.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.800+0000 7f79e7fff640 10 client.?.objecter ms_handle_connect 0x7f7a0014be90 2024-03-25T19:44:37.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.800+0000 7f79e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.800 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.801+0000 7f79e7fff640 10 client.?.objecter ms_handle_connect 0x7f7a000b1730 2024-03-25T19:44:37.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.801+0000 7f79e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.802+0000 7f79e7fff640 10 client.?.objecter ms_handle_connect 0x7f7a000b4d60 2024-03-25T19:44:37.801 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.802+0000 7f79e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643582443 0 0) 0x7f7a00141a30 con 0x7f7a000b1730 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:44:37.814 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.814+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ec004eb0 con 0x7f7a000b1730 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723916900 0 0) 0x7f7a001d7860 con 0x7f7a0014be90 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:44:37.815 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.816+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a00141a30 con 0x7f7a0014be90 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515421058 0 0) 0x7f7a00140d00 con 0x7f7a000b4d60 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:44:37.820 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.820+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a001d7860 con 0x7f7a000b4d60 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3227324470 0 0) 0x7f79ec004eb0 con 0x7f7a000b1730 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a00140d00 con 0x7f7a000b1730 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f00019f0 con 0x7f7a000b1730 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:37.841 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.842 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.841+0000 7f79e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4153518050 0 0) 0x7f7a00141a30 con 0x7f7a0014be90 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79ec004eb0 con 0x7f7a0014be90 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f8001b50 con 0x7f7a0014be90 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.845+0000 7f79e7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:37.845 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:37.846 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:37.846 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4229903634 0 0) 0x7f7a001d7860 con 0x7f7a000b4d60 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a00141a30 con 0x7f7a000b4d60 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f4001b00 con 0x7f7a000b4d60 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:37.857 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.858+0000 7f79e7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:37.858 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1502240300 0 0) 0x7f7a00140d00 con 0x7f7a000b1730 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 >> v1:172.21.15.196:6789/0 conn(0x7f7a000b4d60 legacy=0x7f7a001d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 >> v1:172.21.15.29:6789/0 conn(0x7f7a0014be90 legacy=0x7f7a00134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 1 monclient: found mon.c 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00141ca0 con 0x7f7a000b1730 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f79e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 5 monclient: authenticate success, global_id 4706 2024-03-25T19:44:37.860 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 10 monclient: _renew_subs 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a001d8400 con 0x7f7a000b1730 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 10 monclient: _renew_subs 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a00138150 con 0x7f7a000b1730 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.860+0000 7f7a05ad3640 1 librados: init done 2024-03-25T19:44:37.861 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.861+0000 7f7a05ad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:37.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.861+0000 7f7a05ad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:37.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.861+0000 7f7a05ad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:37.862 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:37.861+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a000627b0 con 0x7f7a000b1730 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79f00047e0 con 0x7f7a000b1730 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f79f0004af0 con 0x7f7a000b1730 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79e7fff640 10 monclient: dump: 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:38.179 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.179+0000 7f79fdffb640 4 set_mon_vals no callback set 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.180+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f79f001ab00 con 0x7f7a000b1730 2024-03-25T19:44:38.180 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.181+0000 7f79feffd640 10 monclient: get_auth_request con 0x7f79ec06e110 auth_method 0 2024-03-25T19:44:38.181 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.181+0000 7f79e7fff640 10 client.4706.objecter ms_handle_connect 0x7f79ec06e110 2024-03-25T19:44:38.187 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f79f00548a0 con 0x7f7a000b1730 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 10 client.4706.objecter ms_dispatch 0x7f7a00000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79f00501e0 con 0x7f7a000b1730 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:38.188 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.188+0000 7f79e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:38.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.412+0000 7f7a05ad3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.412+0000 7f7a05ad3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.412+0000 7f7a05ad3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:38.412 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.412+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a00069f40 con 0x7f7a000b1730 2024-03-25T19:44:38.544 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.544+0000 7f79e7fff640 1 -- 172.21.15.29:0/637533120 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f79f00522e0 con 0x7f7a000b1730 2024-03-25T19:44:38.544 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.544+0000 7f79e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.544 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.544+0000 7f79e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:38.544 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:38.544 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 10 librados: watch_flush enter 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 10 librados: watch_flush exit 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 10 client.4706.objecter successfully canceled tick 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 >> v1:172.21.15.29:6800/2314944989 conn(0x7f79ec06e110 legacy=0x7f79ec0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 10 monclient: shutdown 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 >> v1:172.21.15.29:6790/0 conn(0x7f7a000b1730 legacy=0x7f7a001405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 shutdown_connections 2024-03-25T19:44:38.547 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 >> 172.21.15.29:0/637533120 conn(0x7f7a000617a0 msgr2=0x7f7a000b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:38.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 shutdown_connections 2024-03-25T19:44:38.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 -- 172.21.15.29:0/637533120 wait complete. 2024-03-25T19:44:38.548 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:38.547+0000 7f7a05ad3640 1 librados: shutdown 2024-03-25T19:44:38.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:45.365 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:45.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:45.484 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 10 monclient: build_initial_monmap 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 10 monclient: monmap: 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:45.485933+0000 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:45.485933+0000 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:45.485 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 1 Processor -- start 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 1 -- start start 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.485+0000 7f396bff6640 10 monclient: init 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-b con 0x7f39640b4d60 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-c con 0x7f39640b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-a con 0x7f396414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640af940 con 0x7f396414be90 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640b0a70 con 0x7f39640b1730 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640b0d50 con 0x7f39640b4d60 2024-03-25T19:44:45.487 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:45.488 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.486+0000 7f396bff6640 10 monclient(hunting): authenticate will time out at 6762.239700s 2024-03-25T19:44:45.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.492+0000 7f396a593640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f396414be90 0x7f396414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40500/0 (socket says 172.21.15.29:40500) 2024-03-25T19:44:45.492 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.492+0000 7f396a593640 1 -- 172.21.15.29:0/1188716650 learned_addr learned my addr 172.21.15.29:0/1188716650 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205211342 0 0) 0x7f39640af940 con 0x7f396414be90 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:44:45.499 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.499+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950003520 con 0x7f396414be90 2024-03-25T19:44:45.512 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068062336 0 0) 0x7f3950003520 con 0x7f396414be90 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 >> v1:172.21.15.29:6790/0 conn(0x7f39640b4d60 legacy=0x7f396413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 >> v1:172.21.15.196:6789/0 conn(0x7f39640b1730 legacy=0x7f39640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: _finish_hunting 0 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 monclient: found mon.noname-a 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640b1030 con 0x7f396414be90 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: _finish_auth 0 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: _check_auth_tickets 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f395c001a10 con 0x7f396414be90 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f396bff6640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f396bff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:45.513 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f3968d90640 10 monclient: dump: 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.514 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.512+0000 7f396bff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:45.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f395c001b70 con 0x7f396414be90 2024-03-25T19:44:45.523 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 1 -- 172.21.15.29:0/1188716650 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f395c001f80 con 0x7f396414be90 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f3968d90640 10 monclient: dump: 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:45.524 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f396bff6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:45.525 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.524+0000 7f396bff6640 4 set_mon_vals no callback set 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 10 monclient: shutdown 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 1 -- 172.21.15.29:0/1188716650 >> v1:172.21.15.29:6789/0 conn(0x7f396414be90 legacy=0x7f396414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 1 -- 172.21.15.29:0/1188716650 shutdown_connections 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 1 -- 172.21.15.29:0/1188716650 >> 172.21.15.29:0/1188716650 conn(0x7f39640617a0 msgr2=0x7f3964061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 1 -- 172.21.15.29:0/1188716650 shutdown_connections 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.525+0000 7f396bff6640 1 -- 172.21.15.29:0/1188716650 wait complete. 2024-03-25T19:44:45.527 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 10 monclient: build_initial_monmap 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 10 monclient: monmap: 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 librados: starting msgr at 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 librados: starting objecter 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 Processor -- start 2024-03-25T19:44:45.528 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 -- start start 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 librados: setting wanted keys 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 1 librados: calling monclient init 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.526+0000 7f396bff6640 10 monclient: init 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-c con 0x7f39640b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-b con 0x7f39640b4d60 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient(hunting): picked mon.noname-a con 0x7f396414be90 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3964140380 con 0x7f396414be90 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3964140660 con 0x7f39640b4d60 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3964141e30 con 0x7f39640b1730 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:45.529 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.527+0000 7f396bff6640 10 monclient(hunting): authenticate will time out at 6762.280368s 2024-03-25T19:44:45.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.532+0000 7f3969591640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f39640b4d60 0x7f3964142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:33804/0 (socket says 172.21.15.29:33804) 2024-03-25T19:44:45.531 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.532+0000 7f3969591640 1 -- 172.21.15.29:0/2265669513 learned_addr learned my addr 172.21.15.29:0/2265669513 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:45.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.533+0000 7f395a7fc640 10 client.?.objecter ms_handle_connect 0x7f39640b4d60 2024-03-25T19:44:45.532 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.533+0000 7f395a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 client.?.objecter ms_handle_connect 0x7f39640b1730 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421025356 0 0) 0x7f3964140660 con 0x7f39640b4d60 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:44:45.550 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.550+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3948004eb0 con 0x7f39640b4d60 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47170385 0 0) 0x7f3964141e30 con 0x7f39640b1730 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:44:45.556 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.557+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3964140660 con 0x7f39640b1730 2024-03-25T19:44:45.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.566+0000 7f395a7fc640 10 client.?.objecter ms_handle_connect 0x7f396414be90 2024-03-25T19:44:45.565 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.566+0000 7f395a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.566+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525583872 0 0) 0x7f3948004eb0 con 0x7f39640b4d60 2024-03-25T19:44:45.566 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3964141e30 con 0x7f39640b4d60 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3954001a30 con 0x7f39640b4d60 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:45.567 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:45.568 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.568 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.567+0000 7f395a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:45.574 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66588823 0 0) 0x7f3964140380 con 0x7f396414be90 2024-03-25T19:44:45.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:44:45.575 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.575+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3948004eb0 con 0x7f396414be90 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752838425 0 0) 0x7f3964140660 con 0x7f39640b1730 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3964140380 con 0x7f39640b1730 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3960001930 con 0x7f39640b1730 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.581+0000 7f395a7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.581 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:45.582 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3908686730 0 0) 0x7f3964140380 con 0x7f39640b1730 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 >> v1:172.21.15.196:6789/0 conn(0x7f39640b4d60 legacy=0x7f3964142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 >> v1:172.21.15.29:6789/0 conn(0x7f396414be90 legacy=0x7f39641e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 1 monclient: found mon.c 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:45.596 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964142110 con 0x7f39640b1730 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f395a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 5 monclient: authenticate success, global_id 4712 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 10 monclient: _renew_subs 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3964140e60 con 0x7f39640b1730 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 10 client.4712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 10 monclient: _renew_subs 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3964142c40 con 0x7f39640b1730 2024-03-25T19:44:45.597 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.596+0000 7f396bff6640 1 librados: init done 2024-03-25T19:44:45.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.597+0000 7f396bff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.597+0000 7f396bff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.597+0000 7f396bff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:45.598 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.597+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39640627b0 con 0x7f39640b1730 2024-03-25T19:44:45.914 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3960003610 con 0x7f39640b1730 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f3960003960 con 0x7f39640b1730 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395a7fc640 10 monclient: dump: 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:45.915 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.914+0000 7f395bfff640 4 set_mon_vals no callback set 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.915+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f3960003b80 con 0x7f39640b1730 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.916+0000 7f3969591640 10 monclient: get_auth_request con 0x7f394806df50 auth_method 0 2024-03-25T19:44:45.916 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.916+0000 7f395a7fc640 10 client.4712.objecter ms_handle_connect 0x7f394806df50 2024-03-25T19:44:45.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f3960053600 con 0x7f39640b1730 2024-03-25T19:44:45.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 10 client.4712.objecter ms_dispatch 0x7f3964000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:45.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 3 client.4712.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:45.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 3 client.4712.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:45.924 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 20 client.4712.objecter dump_active .. 0 homeless 2024-03-25T19:44:45.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.924+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f396004f110 con 0x7f39640b1730 2024-03-25T19:44:45.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.925+0000 7f395a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.925 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:45.925+0000 7f395a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:46.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.147+0000 7f396bff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.147+0000 7f396bff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.147+0000 7f396bff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:44:46.147 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.147+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39640b5cf0 con 0x7f39640b1730 2024-03-25T19:44:46.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.170+0000 7f395a7fc640 1 -- 172.21.15.29:0/2265669513 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f3960051040 con 0x7f39640b1730 2024-03-25T19:44:46.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.170+0000 7f395a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.170 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.170+0000 7f395a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:46.170 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:46.170 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 10 librados: watch_flush enter 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 10 librados: watch_flush exit 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 20 client.4712.objecter shutdown clearing up homeless session... 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 10 client.4712.objecter successfully canceled tick 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 >> v1:172.21.15.29:6800/2314944989 conn(0x7f394806df50 legacy=0x7f3948070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.173 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 10 monclient: shutdown 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 >> v1:172.21.15.29:6790/0 conn(0x7f39640b1730 legacy=0x7f3964138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 shutdown_connections 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 >> 172.21.15.29:0/2265669513 conn(0x7f39640617a0 msgr2=0x7f39640b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 shutdown_connections 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 -- 172.21.15.29:0/2265669513 wait complete. 2024-03-25T19:44:46.174 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:46.173+0000 7f396bff6640 1 librados: shutdown 2024-03-25T19:44:46.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:51.708 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:21.871 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.711 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:21.874 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.720 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:21.884 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.746 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:21.909 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.767 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:21.929 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.778 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:21.940 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.779 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:21.943 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.797 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:21.961 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.801 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:21.964 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.968 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:22.130 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.973 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:22.135 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.037 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:22.199 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.076 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:22.240 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.080 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:22.243 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.098 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:22.262 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.106 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:22.270 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.108 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:22.272 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.153 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:22.316 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:52.991 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:53.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.098+0000 7fbfe2e15640 10 monclient: get_monmap_and_config 2024-03-25T19:44:53.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.099+0000 7fbfe2e15640 10 monclient: build_initial_monmap 2024-03-25T19:44:53.098 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.099+0000 7fbfe2e15640 10 monclient: monmap: 2024-03-25T19:44:53.098 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:53.098 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:44:53.099817+0000 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:44:53.099817+0000 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:44:53.099 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.099+0000 7fbfe2e15640 1 Processor -- start 2024-03-25T19:44:53.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 1 -- start start 2024-03-25T19:44:53.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient: init 2024-03-25T19:44:53.100 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc1430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc142520 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0b28a0 con 0x7fbfdc13f710 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0b2a30 con 0x7fbfdc1430c0 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:53.101 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.100+0000 7fbfe2e15640 10 monclient(hunting): authenticate will time out at 6769.853660s 2024-03-25T19:44:53.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.105+0000 7fbfe0bb1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbfdc0a98f0 0x7fbfdc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54122/0 (socket says 172.21.15.29:54122) 2024-03-25T19:44:53.105 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.105+0000 7fbfe0bb1640 1 -- 172.21.15.29:0/4027955922 learned_addr learned my addr 172.21.15.29:0/4027955922 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.114+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609955273 0 0) 0x7fbfdc142520 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.115+0000 7fbfdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.115+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.115+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.115+0000 7fbfdb7fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:44:53.114 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.115+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfcc003520 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848670230 0 0) 0x7fbfdc0b2a30 con 0x7fbfdc1430c0 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:44:53.116 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.117+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc142520 con 0x7fbfdc1430c0 2024-03-25T19:44:53.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399566877 0 0) 0x7fbfdc0b28a0 con 0x7fbfdc13f710 2024-03-25T19:44:53.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.118 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:44:53.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.119+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc0b2a30 con 0x7fbfdc13f710 2024-03-25T19:44:53.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757700816 0 0) 0x7fbfcc003520 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.127 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 >> v1:172.21.15.29:6790/0 conn(0x7fbfdc1430c0 legacy=0x7fbfdc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 >> v1:172.21.15.196:6789/0 conn(0x7fbfdc13f710 legacy=0x7fbfdc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 monclient: found mon.noname-a 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc0b44c0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbfd0001690 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:53.128 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfdb7fe640 10 monclient: dump: 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfe2e15640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:44:53.129 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.125+0000 7fbfe2e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfd00017f0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 1 -- 172.21.15.29:0/4027955922 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbfd0001c00 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfe2e15640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfe2e15640 4 set_mon_vals no callback set 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.136+0000 7fbfdb7fe640 10 monclient: dump: 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.136 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:53.137 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.138 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 10 monclient: shutdown 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 1 -- 172.21.15.29:0/4027955922 >> v1:172.21.15.29:6789/0 conn(0x7fbfdc0a98f0 legacy=0x7fbfdc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 1 -- 172.21.15.29:0/4027955922 shutdown_connections 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 1 -- 172.21.15.29:0/4027955922 >> 172.21.15.29:0/4027955922 conn(0x7fbfdc0b7ea0 msgr2=0x7fbfdc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 1 -- 172.21.15.29:0/4027955922 shutdown_connections 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.137+0000 7fbfe2e15640 1 -- 172.21.15.29:0/4027955922 wait complete. 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 10 monclient: build_initial_monmap 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 10 monclient: monmap: 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:53.139 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 librados: starting msgr at 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 librados: starting objecter 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 Processor -- start 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 -- start start 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 librados: setting wanted keys 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 1 librados: calling monclient init 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.138+0000 7fbfe2e15640 10 monclient: init 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc0a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc13f710 addr v1:172.21.15.29:6790/0 2024-03-25T19:44:53.140 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0b7ce0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc1319b0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0b7800 con 0x7fbfdc13f710 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.139+0000 7fbfe2e15640 10 monclient(hunting): authenticate will time out at 6769.892344s 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.140+0000 7fbfe13b2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbfdc1430c0 0x7fbfdc1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:36286/0 (socket says 172.21.15.29:36286) 2024-03-25T19:44:53.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.140+0000 7fbfe13b2640 1 -- 172.21.15.29:0/2020706684 learned_addr learned my addr 172.21.15.29:0/2020706684 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:44:53.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.142+0000 7fbfd97fa640 10 client.?.objecter ms_handle_connect 0x7fbfdc1430c0 2024-03-25T19:44:53.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.142+0000 7fbfd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736516309 0 0) 0x7fbfdc1319b0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0004eb0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 client.?.objecter ms_handle_connect 0x7fbfdc0a98f0 2024-03-25T19:44:53.150 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.150+0000 7fbfd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:53.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.157+0000 7fbfd97fa640 10 client.?.objecter ms_handle_connect 0x7fbfdc13f710 2024-03-25T19:44:53.157 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.157+0000 7fbfd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:53.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044522323 0 0) 0x7fbfdc0b7ce0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.160 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:44:53.161 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.161+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc1319b0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076852335 0 0) 0x7fbfdc0b7800 con 0x7fbfdc13f710 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:44:53.164 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.164+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc0b7ce0 con 0x7fbfdc13f710 2024-03-25T19:44:53.165 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4137572210 0 0) 0x7fbfc0004eb0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfdc0b7800 con 0x7fbfdc1430c0 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbfd4001c90 con 0x7fbfdc1430c0 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 10 monclient(hunting): dump: 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:53.166 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:53.167 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:53.167 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.167 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.166+0000 7fbfd97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2853489047 0 0) 0x7fbfdc1319b0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc0004eb0 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbfd0001b00 con 0x7fbfdc0a98f0 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.168+0000 7fbfd97fa640 10 monclient(hunting): dump: 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:53.168 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:53.169 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3539877334 0 0) 0x7fbfdc0b7800 con 0x7fbfdc1430c0 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 >> v1:172.21.15.29:6790/0 conn(0x7fbfdc13f710 legacy=0x7fbfdc130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 >> v1:172.21.15.29:6789/0 conn(0x7fbfdc0a98f0 legacy=0x7fbfdc1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 1 monclient: found mon.b 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc136f50 con 0x7fbfdc1430c0 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:53.177 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 10 monclient: _renew_subs 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc1350f0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 10 monclient: _renew_subs 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc135c60 con 0x7fbfdc1430c0 2024-03-25T19:44:53.178 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.177+0000 7fbfe2e15640 1 librados: init done 2024-03-25T19:44:53.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.178+0000 7fbfe2e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.178+0000 7fbfe2e15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.178+0000 7fbfe2e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:53.179 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.178+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc0627b0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.396 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.397+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfd4003970 con 0x7fbfdc1430c0 2024-03-25T19:44:53.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.397+0000 7fbfd97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:53.397 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.397+0000 7fbfdaffd640 4 set_mon_vals no callback set 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.460+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbfd4003cc0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.460+0000 7fbfd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.460+0000 7fbfd97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.460+0000 7fbfd97fa640 10 monclient: dump: 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.460 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.462+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbfd4003ea0 con 0x7fbfdc1430c0 2024-03-25T19:44:53.461 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.462+0000 7fbfe0bb1640 10 monclient: get_auth_request con 0x7fbfc006df70 auth_method 0 2024-03-25T19:44:53.462 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.462+0000 7fbfd97fa640 10 client.4723.objecter ms_handle_connect 0x7fbfc006df70 2024-03-25T19:44:53.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.464+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fbfd4054b10 con 0x7fbfdc1430c0 2024-03-25T19:44:53.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.464+0000 7fbfd97fa640 10 client.4723.objecter ms_dispatch 0x7fbfdc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:53.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.464+0000 7fbfd97fa640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:53.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.464+0000 7fbfd97fa640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:53.464 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.464+0000 7fbfd97fa640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T19:44:53.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.468+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfd4050620 con 0x7fbfdc1430c0 2024-03-25T19:44:53.467 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.468+0000 7fbfd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.468 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.468+0000 7fbfd97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:53.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.725+0000 7fbfe2e15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.725+0000 7fbfe2e15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.725+0000 7fbfe2e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:44:53.725 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.725+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfdc13e270 con 0x7fbfdc1430c0 2024-03-25T19:44:53.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.759+0000 7fbfd97fa640 1 -- 172.21.15.29:0/2020706684 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fbfd4052550 con 0x7fbfdc1430c0 2024-03-25T19:44:53.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.759+0000 7fbfd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.759 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.759+0000 7fbfd97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:53.759 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:44:53.759 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 10 librados: watch_flush enter 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 10 librados: watch_flush exit 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 10 client.4723.objecter successfully canceled tick 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbfc006df70 legacy=0x7fbfc0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.761 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 10 monclient: shutdown 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 >> v1:172.21.15.196:6789/0 conn(0x7fbfdc1430c0 legacy=0x7fbfdc1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 shutdown_connections 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 >> 172.21.15.29:0/2020706684 conn(0x7fbfdc0b7ea0 msgr2=0x7fbfdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 shutdown_connections 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 -- 172.21.15.29:0/2020706684 wait complete. 2024-03-25T19:44:53.762 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:44:53.761+0000 7fbfe2e15640 1 librados: shutdown 2024-03-25T19:44:53.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:00.580 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.681+0000 7fbb75196640 10 monclient: get_monmap_and_config 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.681+0000 7fbb75196640 10 monclient: build_initial_monmap 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.681+0000 7fbb75196640 10 monclient: monmap: 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:45:00.682438+0000 2024-03-25T19:45:00.681 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:45:00.682438+0000 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:45:00.682 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.682+0000 7fbb75196640 1 Processor -- start 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.682+0000 7fbb75196640 1 -- start start 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.682+0000 7fbb75196640 10 monclient: init 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:00.683 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-b con 0x7fbb700a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-a con 0x7fbb701430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70142520 con 0x7fbb701430c0 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700b28a0 con 0x7fbb7013f710 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700b2a30 con 0x7fbb700a98f0 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:00.684 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.683+0000 7fbb75196640 10 monclient(hunting): authenticate will time out at 6777.436115s 2024-03-25T19:45:00.685 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.686+0000 7fbb6e59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbb7013f710 0x7fbb701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:39592/0 (socket says 172.21.15.29:39592) 2024-03-25T19:45:00.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.686+0000 7fbb6e59c640 1 -- 172.21.15.29:0/3127585708 learned_addr learned my addr 172.21.15.29:0/3127585708 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.693+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827667388 0 0) 0x7fbb700b28a0 con 0x7fbb7013f710 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.693+0000 7fbb6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.693+0000 7fbb6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.693+0000 7fbb6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.693+0000 7fbb6dd9b640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:45:00.693 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.694+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58003520 con 0x7fbb7013f710 2024-03-25T19:45:00.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036113083 0 0) 0x7fbb700b2a30 con 0x7fbb700a98f0 2024-03-25T19:45:00.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.702 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb700b28a0 con 0x7fbb700a98f0 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3080414530 0 0) 0x7fbb58003520 con 0x7fbb7013f710 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 >> v1:172.21.15.29:6790/0 conn(0x7fbb700a98f0 legacy=0x7fbb700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 >> v1:172.21.15.29:6789/0 conn(0x7fbb701430c0 legacy=0x7fbb7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700b44c0 con 0x7fbb7013f710 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbb64001690 con 0x7fbb7013f710 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.703 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb6dd9b640 10 monclient: dump: 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb75196640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:45:00.704 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.702+0000 7fbb75196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb640018b0 con 0x7fbb7013f710 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 1 -- 172.21.15.29:0/3127585708 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbb64001c40 con 0x7fbb7013f710 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb75196640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb75196640 4 set_mon_vals no callback set 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.710 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.710+0000 7fbb6dd9b640 10 monclient: dump: 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:00.711 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 10 monclient: shutdown 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 1 -- 172.21.15.29:0/3127585708 >> v1:172.21.15.196:6789/0 conn(0x7fbb7013f710 legacy=0x7fbb701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 1 -- 172.21.15.29:0/3127585708 shutdown_connections 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 1 -- 172.21.15.29:0/3127585708 >> 172.21.15.29:0/3127585708 conn(0x7fbb700b7ea0 msgr2=0x7fbb700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 1 -- 172.21.15.29:0/3127585708 shutdown_connections 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.711+0000 7fbb75196640 1 -- 172.21.15.29:0/3127585708 wait complete. 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.712+0000 7fbb75196640 10 monclient: build_initial_monmap 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.712+0000 7fbb75196640 10 monclient: monmap: 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:45:00.714 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 librados: starting msgr at 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 librados: starting objecter 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 Processor -- start 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 -- start start 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 librados: setting wanted keys 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 1 librados: calling monclient init 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 10 monclient: init 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.713+0000 7fbb75196640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:00.715 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-a con 0x7fbb700a98f0 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-b con 0x7fbb7013f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 10 monclient(hunting): picked mon.noname-c con 0x7fbb701430c0 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb701d77a0 con 0x7fbb700a98f0 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70138e40 con 0x7fbb7013f710 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700aba30 con 0x7fbb701430c0 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:00.716 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.714+0000 7fbb75196640 10 monclient(hunting): authenticate will time out at 6777.467121s 2024-03-25T19:45:00.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.724+0000 7fbb6e59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbb7013f710 0x7fbb701d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:39626/0 (socket says 172.21.15.29:39626) 2024-03-25T19:45:00.723 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.724+0000 7fbb6e59c640 1 -- 172.21.15.29:0/637123731 learned_addr learned my addr 172.21.15.29:0/637123731 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:00.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.725+0000 7fbb577fe640 10 client.?.objecter ms_handle_connect 0x7fbb7013f710 2024-03-25T19:45:00.724 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.725+0000 7fbb577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.726 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.727+0000 7fbb577fe640 10 client.?.objecter ms_handle_connect 0x7fbb701430c0 2024-03-25T19:45:00.727 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.727+0000 7fbb577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875751322 0 0) 0x7fbb70138e40 con 0x7fbb7013f710 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:45:00.732 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.732+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c004eb0 con 0x7fbb7013f710 2024-03-25T19:45:00.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.737+0000 7fbb577fe640 10 client.?.objecter ms_handle_connect 0x7fbb700a98f0 2024-03-25T19:45:00.736 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.737+0000 7fbb577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599982477 0 0) 0x7fbb700aba30 con 0x7fbb701430c0 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.740+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb70138e40 con 0x7fbb701430c0 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248825036 0 0) 0x7fbb5c004eb0 con 0x7fbb7013f710 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb700aba30 con 0x7fbb7013f710 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbb64001690 con 0x7fbb7013f710 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): dump: 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.744 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115482624 0 0) 0x7fbb701d77a0 con 0x7fbb700a98f0 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:45:00.745 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.744+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c004eb0 con 0x7fbb700a98f0 2024-03-25T19:45:00.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874084417 0 0) 0x7fbb70138e40 con 0x7fbb701430c0 2024-03-25T19:45:00.749 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb701d77a0 con 0x7fbb701430c0 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbb68001b10 con 0x7fbb701430c0 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.750+0000 7fbb577fe640 10 monclient(hunting): dump: 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:00.750 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:00.754 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1104030738 0 0) 0x7fbb701d77a0 con 0x7fbb701430c0 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 >> v1:172.21.15.196:6789/0 conn(0x7fbb7013f710 legacy=0x7fbb701d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 >> v1:172.21.15.29:6789/0 conn(0x7fbb700a98f0 legacy=0x7fbb70138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 1 monclient: found mon.c 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb701d8340 con 0x7fbb701430c0 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 10 monclient: _renew_subs 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb70134820 con 0x7fbb701430c0 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 10 monclient: _renew_subs 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb701380c0 con 0x7fbb701430c0 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.754+0000 7fbb75196640 1 librados: init done 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.756+0000 7fbb75196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:00.756 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.756+0000 7fbb75196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:00.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.756+0000 7fbb75196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:45:00.757 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:00.756+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb700627b0 con 0x7fbb701430c0 2024-03-25T19:45:01.071 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb68002000 con 0x7fbb701430c0 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fbb68003b30 con 0x7fbb701430c0 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb577fe640 10 monclient: dump: 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:01.072 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.071+0000 7fbb6d59a640 4 set_mon_vals no callback set 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.072+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fbb68003d50 con 0x7fbb701430c0 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.073+0000 7fbb6ed9d640 10 monclient: get_auth_request con 0x7fbb5c06df50 auth_method 0 2024-03-25T19:45:01.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.073+0000 7fbb577fe640 10 client.4724.objecter ms_handle_connect 0x7fbb5c06df50 2024-03-25T19:45:01.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.078+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fbb68053810 con 0x7fbb701430c0 2024-03-25T19:45:01.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.078+0000 7fbb577fe640 10 client.4724.objecter ms_dispatch 0x7fbb70000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:01.078 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.078+0000 7fbb577fe640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:01.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.078+0000 7fbb577fe640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:01.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.079+0000 7fbb577fe640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T19:45:01.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.079+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb6804f320 con 0x7fbb701430c0 2024-03-25T19:45:01.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.079+0000 7fbb577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:01.079 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.079+0000 7fbb577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:01.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.336+0000 7fbb75196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.336+0000 7fbb75196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.336+0000 7fbb75196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.29:6790/0 2024-03-25T19:45:01.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.337+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 --> v1:172.21.15.29:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb7014e880 con 0x7fbb701430c0 2024-03-25T19:45:01.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.378+0000 7fbb577fe640 1 -- 172.21.15.29:0/637123731 <== mon.2 v1:172.21.15.29:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fbb68051250 con 0x7fbb701430c0 2024-03-25T19:45:01.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.378+0000 7fbb577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.378 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.378+0000 7fbb577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:01.378 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:45:01.378 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:01.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 10 librados: watch_flush enter 2024-03-25T19:45:01.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 10 librados: watch_flush exit 2024-03-25T19:45:01.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T19:45:01.381 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 10 client.4724.objecter successfully canceled tick 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 >> v1:172.21.15.29:6800/2314944989 conn(0x7fbb5c06df50 legacy=0x7fbb5c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 10 monclient: shutdown 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 >> v1:172.21.15.29:6790/0 conn(0x7fbb701430c0 legacy=0x7fbb70134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 shutdown_connections 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 >> 172.21.15.29:0/637123731 conn(0x7fbb700b7ea0 msgr2=0x7fbb70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 shutdown_connections 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 -- 172.21.15.29:0/637123731 wait complete. 2024-03-25T19:45:01.382 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:01.381+0000 7fbb75196640 1 librados: shutdown 2024-03-25T19:45:01.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:08.199 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.304+0000 7f9080c25640 10 monclient: get_monmap_and_config 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.304+0000 7f9080c25640 10 monclient: build_initial_monmap 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.304+0000 7f9080c25640 10 monclient: monmap: 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:45:08.305348+0000 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:45:08.305348+0000 2024-03-25T19:45:08.304 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:08.305 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:08.305 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:08.305 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:45:08.305 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:45:08.305 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.305+0000 7f9080c25640 1 Processor -- start 2024-03-25T19:45:08.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.305+0000 7f9080c25640 1 -- start start 2024-03-25T19:45:08.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.305+0000 7f9080c25640 10 monclient: init 2024-03-25T19:45:08.306 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-b con 0x7f907c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-a con 0x7f907c13f710 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-c con 0x7f907c1430c0 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c142520 con 0x7f907c13f710 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c0b28a0 con 0x7f907c1430c0 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c0b2a30 con 0x7f907c0a98f0 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:08.307 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.306+0000 7f9080c25640 10 monclient(hunting): authenticate will time out at 6785.059495s 2024-03-25T19:45:08.310 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.311+0000 7f907a59c640 1 --1- >> v1:172.21.15.29:6790/0 conn(0x7f907c0a98f0 0x7f907c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6790/0 says I am v1:172.21.15.29:59460/0 (socket says 172.21.15.29:59460) 2024-03-25T19:45:08.311 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.311+0000 7f907a59c640 1 -- 172.21.15.29:0/4106269056 learned_addr learned my addr 172.21.15.29:0/4106269056 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:08.322 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173153154 0 0) 0x7f907c0b28a0 con 0x7f907c1430c0 2024-03-25T19:45:08.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:45:08.323 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.323+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9064003520 con 0x7f907c1430c0 2024-03-25T19:45:08.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865658817 0 0) 0x7f907c142520 con 0x7f907c13f710 2024-03-25T19:45:08.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.333 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:45:08.334 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.333+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f907c0b28a0 con 0x7f907c13f710 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625988364 0 0) 0x7f9064003520 con 0x7f907c1430c0 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 >> v1:172.21.15.29:6790/0 conn(0x7f907c0a98f0 legacy=0x7f907c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 >> v1:172.21.15.29:6789/0 conn(0x7f907c13f710 legacy=0x7f907c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 monclient: found mon.noname-c 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0b44c0 con 0x7f907c1430c0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: _finish_auth 0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9074001a10 con 0x7f907c1430c0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f907959a640 10 monclient: dump: 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f9080c25640 5 monclient: authenticate success, global_id 4732 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.336+0000 7f9080c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9074001b70 con 0x7f907c1430c0 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 1 -- 172.21.15.29:0/4106269056 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9074001f80 con 0x7f907c1430c0 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f9080c25640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f9080c25640 4 set_mon_vals no callback set 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.354+0000 7f907959a640 10 monclient: dump: 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:08.354 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 10 monclient: shutdown 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 1 -- 172.21.15.29:0/4106269056 >> v1:172.21.15.196:6789/0 conn(0x7f907c1430c0 legacy=0x7f907c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 1 -- 172.21.15.29:0/4106269056 shutdown_connections 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 1 -- 172.21.15.29:0/4106269056 >> 172.21.15.29:0/4106269056 conn(0x7f907c0b7ea0 msgr2=0x7f907c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 1 -- 172.21.15.29:0/4106269056 shutdown_connections 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.355+0000 7f9080c25640 1 -- 172.21.15.29:0/4106269056 wait complete. 2024-03-25T19:45:08.356 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.356+0000 7f9080c25640 10 monclient: build_initial_monmap 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.356+0000 7f9080c25640 10 monclient: monmap: 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.356+0000 7f9080c25640 1 librados: starting msgr at 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.356+0000 7f9080c25640 1 librados: starting objecter 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.356+0000 7f9080c25640 1 Processor -- start 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 -- start start 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 librados: setting wanted keys 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 librados: calling monclient init 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient: init 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-c con 0x7f907c0a98f0 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-b con 0x7f907c13f710 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient(hunting): picked mon.noname-a con 0x7f907c1430c0 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c06eb40 con 0x7f907c1430c0 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c133730 con 0x7f907c13f710 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c06edb0 con 0x7f907c0a98f0 2024-03-25T19:45:08.359 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:08.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.357+0000 7f9080c25640 10 monclient(hunting): authenticate will time out at 6785.110765s 2024-03-25T19:45:08.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.361+0000 7f9079d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f907c13f710 0x7f907c132910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:46360/0 (socket says 172.21.15.29:46360) 2024-03-25T19:45:08.360 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.361+0000 7f9079d9b640 1 -- 172.21.15.29:0/2123195151 learned_addr learned my addr 172.21.15.29:0/2123195151 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:08.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.361+0000 7f906affd640 10 client.?.objecter ms_handle_connect 0x7f907c13f710 2024-03-25T19:45:08.361 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.361+0000 7f906affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.368 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.369+0000 7f906affd640 10 client.?.objecter ms_handle_connect 0x7f907c0a98f0 2024-03-25T19:45:08.369 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.369+0000 7f906affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045191408 0 0) 0x7f907c133730 con 0x7f907c13f710 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.371+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c004eb0 con 0x7f907c13f710 2024-03-25T19:45:08.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.374+0000 7f906affd640 10 client.?.objecter ms_handle_connect 0x7f907c1430c0 2024-03-25T19:45:08.373 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.374+0000 7f906affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234112419 0 0) 0x7f905c004eb0 con 0x7f907c13f710 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f907c133730 con 0x7f907c13f710 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9070001a90 con 0x7f907c13f710 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 10 monclient(hunting): dump: 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.392 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.393 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.392+0000 7f906affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:08.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413477391 0 0) 0x7f907c06edb0 con 0x7f907c0a98f0 2024-03-25T19:45:08.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.394 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:45:08.395 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.395+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c004eb0 con 0x7f907c0a98f0 2024-03-25T19:45:08.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190939817 0 0) 0x7f907c133730 con 0x7f907c13f710 2024-03-25T19:45:08.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 >> v1:172.21.15.29:6790/0 conn(0x7f907c0a98f0 legacy=0x7f907c06f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 >> v1:172.21.15.29:6789/0 conn(0x7f907c1430c0 legacy=0x7f907c133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 1 monclient: found mon.b 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c06f090 con 0x7f907c13f710 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 10 monclient: _finish_auth 0 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f906affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 10 monclient: _renew_subs 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c06f3d0 con 0x7f907c13f710 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 10 monclient: _renew_subs 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.404 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.403+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c13c170 con 0x7f907c13f710 2024-03-25T19:45:08.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.404+0000 7f9080c25640 1 librados: init done 2024-03-25T19:45:08.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.404+0000 7f9080c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.404+0000 7f9080c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.404+0000 7f9080c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.405 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.404+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c06ff00 con 0x7f907c13f710 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9070001d10 con 0x7f907c13f710 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7f9070003a60 con 0x7f907c13f710 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:08.426 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f9078d99640 4 set_mon_vals no callback set 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.426+0000 7f906affd640 10 monclient: dump: 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:08.427 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:08.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.427+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7f9070003c40 con 0x7f907c13f710 2024-03-25T19:45:08.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.428+0000 7f907a59c640 10 monclient: get_auth_request con 0x7f905c06deb0 auth_method 0 2024-03-25T19:45:08.428 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.428+0000 7f906affd640 10 client.4735.objecter ms_handle_connect 0x7f905c06deb0 2024-03-25T19:45:08.435 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.436+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7f9070054900 con 0x7f907c13f710 2024-03-25T19:45:08.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.436+0000 7f906affd640 10 client.4735.objecter ms_dispatch 0x7f907c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:08.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.436+0000 7f906affd640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:08.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.436+0000 7f906affd640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:08.436 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.436+0000 7f906affd640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:45:08.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.706+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9070050240 con 0x7f907c13f710 2024-03-25T19:45:08.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.707+0000 7f906affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.706 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.707+0000 7f906affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.970+0000 7f9080c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.970+0000 7f9080c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.970+0000 7f9080c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.970+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f907c06f220 con 0x7f907c13f710 2024-03-25T19:45:08.992 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.993+0000 7f906affd640 1 -- 172.21.15.29:0/2123195151 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7f9070052340 con 0x7f907c13f710 2024-03-25T19:45:08.993 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.993+0000 7f906affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.993 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.993+0000 7f906affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:08.993 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:45:08.993 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 10 librados: watch_flush enter 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 10 librados: watch_flush exit 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 >> v1:172.21.15.29:6800/2314944989 conn(0x7f905c06deb0 legacy=0x7f905c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 10 monclient: shutdown 2024-03-25T19:45:08.996 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 >> v1:172.21.15.196:6789/0 conn(0x7f907c13f710 legacy=0x7f907c132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 shutdown_connections 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 >> 172.21.15.29:0/2123195151 conn(0x7f907c0b7ea0 msgr2=0x7f907c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 shutdown_connections 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 -- 172.21.15.29:0/2123195151 wait complete. 2024-03-25T19:45:08.997 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:08.996+0000 7f9080c25640 1 librados: shutdown 2024-03-25T19:45:09.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:11.737 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:41.901 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.744 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:41.908 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.781 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:41.944 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.798 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:41.960 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.809 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:41.973 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.810 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:41.972 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.817 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:41.981 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.845 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:11:42.008 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.846 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:42.010 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:15.814 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.931+0000 7fe052b0e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.931+0000 7fe052b0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.931+0000 7fe052b0e640 10 monclient: monmap: 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:45:15.932383+0000 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:45:15.932383+0000 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:15.931 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:15.932 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:15.932 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-25T19:45:15.932 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-b 2024-03-25T19:45:15.932 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 1 Processor -- start 2024-03-25T19:45:15.933 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 1 -- start start 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient: init 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe04c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe04c0b1730 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.932+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe04c14be90 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe052b0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c0af940 con 0x7fe04c0b4d60 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe052b0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c0b0a70 con 0x7fe04c14be90 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe052b0e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c0b0d50 con 0x7fe04c0b1730 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe052b0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe052b0e640 10 monclient(hunting): authenticate will time out at 6792.685913s 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe0510ab640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe04c14be90 0x7fe04c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.29:50966/0 (socket says 172.21.15.29:50966) 2024-03-25T19:45:15.934 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:15.933+0000 7fe0510ab640 1 -- 172.21.15.29:0/3097111125 learned_addr learned my addr 172.21.15.29:0/3097111125 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:16.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767149028 0 0) 0x7fe04c0af940 con 0x7fe04c0b4d60 2024-03-25T19:45:16.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:16.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:16.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:16.010 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:45:16.011 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.010+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe03c003520 con 0x7fe04c0b4d60 2024-03-25T19:45:16.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202007479 0 0) 0x7fe03c003520 con 0x7fe04c0b4d60 2024-03-25T19:45:16.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 >> v1:172.21.15.29:6790/0 conn(0x7fe04c0b1730 legacy=0x7fe04c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 >> v1:172.21.15.196:6789/0 conn(0x7fe04c14be90 legacy=0x7fe04c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:16.028 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.29:6789/0 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0b1030 con 0x7fe04c0b4d60 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe040001690 con 0x7fe04c0b4d60 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe04b7fe640 10 monclient: dump: 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.029 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe052b0e640 5 monclient: authenticate success, global_id 4740 2024-03-25T19:45:16.030 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.028+0000 7fe052b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:16.045 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 <== mon.0 v1:172.21.15.29:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0400017f0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 1 -- 172.21.15.29:0/3097111125 <== mon.0 v1:172.21.15.29:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe040001c00 con 0x7fe04c0b4d60 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe052b0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe052b0e640 4 set_mon_vals no callback set 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.046+0000 7fe04b7fe640 10 monclient: dump: 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.046 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.047 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 10 monclient: shutdown 2024-03-25T19:45:16.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:16.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 1 -- 172.21.15.29:0/3097111125 >> v1:172.21.15.29:6789/0 conn(0x7fe04c0b4d60 legacy=0x7fe04c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 1 -- 172.21.15.29:0/3097111125 shutdown_connections 2024-03-25T19:45:16.049 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 1 -- 172.21.15.29:0/3097111125 >> 172.21.15.29:0/3097111125 conn(0x7fe04c0617a0 msgr2=0x7fe04c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 1 -- 172.21.15.29:0/3097111125 shutdown_connections 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.047+0000 7fe052b0e640 1 -- 172.21.15.29:0/3097111125 wait complete. 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 10 monclient: monmap: 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a 2024-03-25T19:45:16.050 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.noname-c 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 librados: starting msgr at 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 librados: starting objecter 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 Processor -- start 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 -- start start 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 librados: setting wanted keys 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 1 librados: calling monclient init 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.048+0000 7fe052b0e640 10 monclient: init 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe04c0b1730 addr v1:172.21.15.196:6789/0 2024-03-25T19:45:16.051 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe04c0b4d60 addr v1:172.21.15.29:6789/0 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe04c14be90 addr v1:172.21.15.29:6790/0 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c13bde0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c140f50 con 0x7fe04c0b1730 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 1 -- --> v1:172.21.15.29:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c1410e0 con 0x7fe04c14be90 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:16.052 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.049+0000 7fe052b0e640 10 monclient(hunting): authenticate will time out at 6792.802650s 2024-03-25T19:45:16.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.067+0000 7fe04bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe04c0b4d60 0x7fe04c1d7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35734/0 (socket says 172.21.15.29:35734) 2024-03-25T19:45:16.066 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.067+0000 7fe04bfff640 1 -- 172.21.15.29:0/456621299 learned_addr learned my addr 172.21.15.29:0/456621299 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-03-25T19:45:16.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.069+0000 7fe0497fa640 10 client.?.objecter ms_handle_connect 0x7fe04c0b4d60 2024-03-25T19:45:16.068 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.069+0000 7fe0497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:16.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.073+0000 7fe0497fa640 10 client.?.objecter ms_handle_connect 0x7fe04c0b1730 2024-03-25T19:45:16.073 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.073+0000 7fe0497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:16.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.077+0000 7fe0497fa640 10 client.?.objecter ms_handle_connect 0x7fe04c14be90 2024-03-25T19:45:16.077 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.077+0000 7fe0497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837177694 0 0) 0x7fe04c140f50 con 0x7fe04c0b1730 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:45:16.087 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.087+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe030004eb0 con 0x7fe04c0b1730 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.2 v1:172.21.15.29:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204821308 0 0) 0x7fe04c1410e0 con 0x7fe04c14be90 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:45:16.091 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.091+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.29:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c140f50 con 0x7fe04c14be90 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652527290 0 0) 0x7fe04c13bde0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:45:16.096 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.096+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c1410e0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.106+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2939796869 0 0) 0x7fe030004eb0 con 0x7fe04c0b1730 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04c13bde0 con 0x7fe04c0b1730 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe040001c00 con 0x7fe04c0b1730 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 10 monclient(hunting): dump: 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.107 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.108 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.108 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.107+0000 7fe0497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881254500 0 0) 0x7fe04c1410e0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.29:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe030004eb0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe0380018f0 con 0x7fe04c0b4d60 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.113+0000 7fe0497fa640 10 monclient(hunting): dump: 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.113 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.114 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.114 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.114 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.2 v1:172.21.15.29:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556935949 0 0) 0x7fe04c140f50 con 0x7fe04c14be90 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.29:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04c1410e0 con 0x7fe04c14be90 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.2 v1:172.21.15.29:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe044001b00 con 0x7fe04c14be90 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.115+0000 7fe0497fa640 10 monclient(hunting): dump: 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.115 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.116 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270118107 0 0) 0x7fe04c13bde0 con 0x7fe04c0b1730 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 >> v1:172.21.15.29:6790/0 conn(0x7fe04c14be90 legacy=0x7fe04c142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 >> v1:172.21.15.29:6789/0 conn(0x7fe04c0b4d60 legacy=0x7fe04c1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 1 monclient: found mon.b 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c1d8450 con 0x7fe04c0b1730 2024-03-25T19:45:16.119 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe0497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 5 monclient: authenticate success, global_id 4741 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 10 monclient: _renew_subs 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04c1d86a0 con 0x7fe04c0b1730 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 10 monclient: _renew_subs 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04c1426d0 con 0x7fe04c0b1730 2024-03-25T19:45:16.120 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.119+0000 7fe052b0e640 1 librados: init done 2024-03-25T19:45:16.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.120+0000 7fe052b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.120+0000 7fe052b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.120+0000 7fe052b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:16.121 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.120+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04c1401f0 con 0x7fe04c0b1730 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0400038a0 con 0x7fe04c0b1730 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2530311082 0 0) 0x7fe040003c70 con 0x7fe04c0b1730 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe0497fa640 10 monclient: dump: 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:fsid 82fd6393-df4e-4dec-8d3f-095400b97e2d 2024-03-25T19:45:16.141 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:created 2024-03-25T19:32:58.869386+0000 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 3 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.a 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:2: v1:172.21.15.29:6790/0 mon.c 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr: 2024-03-25T19:45:16.142 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.141+0000 7fe04affd640 4 set_mon_vals no callback set 2024-03-25T19:45:16.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.142+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 376086092 0 0) 0x7fe040003e90 con 0x7fe04c0b1730 2024-03-25T19:45:16.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.143+0000 7fe04bfff640 10 monclient: get_auth_request con 0x7fe03006e0f0 auth_method 0 2024-03-25T19:45:16.143 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.143+0000 7fe0497fa640 10 client.4741.objecter ms_handle_connect 0x7fe03006e0f0 2024-03-25T19:45:16.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.145+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523728526 0 0) 0x7fe040053740 con 0x7fe04c0b1730 2024-03-25T19:45:16.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.145+0000 7fe0497fa640 10 client.4741.objecter ms_dispatch 0x7fe04c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:16.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.145+0000 7fe0497fa640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:16.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.145+0000 7fe0497fa640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:16.145 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.145+0000 7fe0497fa640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T19:45:16.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.403+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe04004f250 con 0x7fe04c0b1730 2024-03-25T19:45:16.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.404+0000 7fe0497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.403 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.404+0000 7fe0497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:16.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.665+0000 7fe052b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.665+0000 7fe052b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.665+0000 7fe052b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-25T19:45:16.665 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.665+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe04c0627b0 con 0x7fe04c0b1730 2024-03-25T19:45:16.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.687+0000 7fe0497fa640 1 -- 172.21.15.29:0/456621299 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1454112059) 0x7fe040051180 con 0x7fe04c0b1730 2024-03-25T19:45:16.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.687+0000 7fe0497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.686 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.687+0000 7fe0497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:16.687 INFO:teuthology.misc.health.smithi029.stdout: 2024-03-25T19:45:16.687 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":11,"fsid":"82fd6393-df4e-4dec-8d3f-095400b97e2d","created":"2024-03-25T19:33:47.207357+0000","modified":"2024-03-25T19:43:51.362146+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:43:51.362146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5739ebf6-47a8-471b-bf92-123ad1d3f33c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"91604cd3-71e6-4c46-ab34-7b0d787ad733","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dc5ee05d-04bc-47b2-9e36-e2bcc0cb9c7f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8396ed08-a86d-47b2-80d4-31af705ae951","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca31575f-d609-411c-97df-72ed610cc543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a6f10ae-99ec-40f1-96a6-9467bde53124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cd5f7e4a-4d35-42f5-96a3-e362c57fd75a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b025c754-d72e-42ca-951d-695966331614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 10 librados: watch_flush enter 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 10 librados: watch_flush exit 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 10 client.4741.objecter successfully canceled tick 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 >> v1:172.21.15.29:6800/2314944989 conn(0x7fe03006e0f0 legacy=0x7fe0300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 10 monclient: shutdown 2024-03-25T19:45:16.690 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 >> v1:172.21.15.196:6789/0 conn(0x7fe04c0b1730 legacy=0x7fe04c140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 shutdown_connections 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 >> 172.21.15.29:0/456621299 conn(0x7fe04c0617a0 msgr2=0x7fe04c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 shutdown_connections 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 -- 172.21.15.29:0/456621299 wait complete. 2024-03-25T19:45:16.691 INFO:teuthology.misc.health.smithi029.stderr:2024-03-25T19:45:16.689+0000 7fe052b0e640 1 librados: shutdown 2024-03-25T19:45:16.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:16.702 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:45:16.703 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:45:16.704 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:45:16.704 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:18.753 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:11:25.960 35383== 2024-03-25T19:45:18.753 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:11:25.960 35383== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:18.790 INFO:tasks.ceph.osd.0.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:22.805 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:45:22.806 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:45:22.806 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:45:22.807 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:24.807 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:11:31.993 35386== 2024-03-25T19:45:24.807 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:11:31.993 35386== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:24.847 INFO:tasks.ceph.osd.1.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:28.909 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:45:28.909 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:45:28.909 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:45:28.909 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:30.904 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:11:38.088 35387== 2024-03-25T19:45:30.904 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:11:38.088 35387== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:30.939 INFO:tasks.ceph.osd.2.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:35.011 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:45:35.012 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:45:35.012 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:45:35.012 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:36.735 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:11:43.931 35385== 2024-03-25T19:45:36.736 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:11:43.931 35385== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:36.768 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:06.931 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.778 INFO:tasks.ceph.osd.3.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:36.818 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:06.981 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.851 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:07.014 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.864 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:07.026 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.867 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:07.031 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.873 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:07.035 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.875 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:07.038 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.900 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:07.063 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:36.909 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:07.072 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:41.114 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:45:41.115 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:45:41.115 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:45:41.115 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:43.021 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:11:50.216 35696== 2024-03-25T19:45:43.021 INFO:tasks.ceph.osd.4.smithi196.stderr:==00:00:11:50.216 35696== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:43.057 INFO:tasks.ceph.osd.4.smithi196.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:47.217 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:45:47.218 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:45:47.218 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:45:47.218 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:49.220 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:11:56.391 35699== 2024-03-25T19:45:49.221 INFO:tasks.ceph.osd.5.smithi196.stderr:==00:00:11:56.391 35699== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:49.263 INFO:tasks.ceph.osd.5.smithi196.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:53.320 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:45:53.320 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:45:53.321 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:45:53.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:45:55.059 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:12:02.231 35698== 2024-03-25T19:45:55.060 INFO:tasks.ceph.osd.6.smithi196.stderr:==00:00:12:02.231 35698== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:45:55.106 INFO:tasks.ceph.osd.6.smithi196.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:45:56.862 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:27.025 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.867 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:27.031 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.884 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:27.047 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.913 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:27.075 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.915 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:27.077 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.927 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:27.090 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.927 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:27.090 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.933 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:27.096 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.950 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:27.113 35074== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:45:59.423 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:45:59.423 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:45:59.424 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:45:59.424 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:00.965 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:12:08.154 35697== 2024-03-25T19:46:00.965 INFO:tasks.ceph.osd.7.smithi196.stderr:==00:00:12:08.154 35697== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:00.994 INFO:tasks.ceph.osd.7.smithi196.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:05.525 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:46:05.526 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:46:05.526 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:46:05.527 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:46:05.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:05.528 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:46:05.528+0000 7f3ab0dc7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35079) UID: 0 2024-03-25T19:46:05.528 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-03-25T19:46:05.528+0000 7f3ab0dc7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:46:05.551 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:46:05.552 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:46:05.552 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:05.578 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:46:05.578 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:46:05.578 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:46:05.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:05.608 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-03-25T19:46:05.596+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35039) UID: 0 2024-03-25T19:46:05.609 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-03-25T19:46:05.598+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:46:08.160 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:38.324 35072== 2024-03-25T19:46:08.160 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:38.324 35072== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:08.170 INFO:tasks.ceph.mon.a.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:11.681 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:46:11.681 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:46:11.682 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:46:11.682 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:11.698 INFO:tasks.ceph.mon.c.smithi029.stderr:2024-03-25T19:46:11.687+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35041) UID: 0 2024-03-25T19:46:11.698 INFO:tasks.ceph.mon.c.smithi029.stderr:2024-03-25T19:46:11.688+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:46:12.243 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:42.406 35074== 2024-03-25T19:46:12.243 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:42.406 35074== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:12.253 INFO:tasks.ceph.mon.c.smithi029.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:17.783 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:46:17.784 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:46:17.784 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:46:17.784 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:17.802 INFO:tasks.ceph.mon.b.smithi196.stderr:2024-03-25T19:46:17.789+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35149) UID: 0 2024-03-25T19:46:17.802 INFO:tasks.ceph.mon.b.smithi196.stderr:2024-03-25T19:46:17.790+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:46:18.242 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:48.404 35153== 2024-03-25T19:46:18.242 INFO:tasks.ceph.mon.b.smithi196.stderr:==00:00:12:48.404 35153== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:18.251 INFO:tasks.ceph.mon.b.smithi196.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:23.886 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi196 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:46:23.886 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:46:23.886 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:46:23.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:46:23.929 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:46:23.929 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:46:24.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:46:24.160 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:46:24.242 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:46:24.243 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:46:24.319 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:46:24.319 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:46:24.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:46:24.391 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:46:24.532 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:46:24.532 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:46:24.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:46:24.610 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:46:24.676 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:46:24.676 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:46:24.735 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:46:24.737 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/data/mon.a.tgz 2024-03-25T19:46:24.737 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-25T19:46:24.753 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.l0faV9jRpR 2024-03-25T19:46:24.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.l0faV9jRpR 2024-03-25T19:46:24.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.l0faV9jRpR 2024-03-25T19:46:24.948 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.l0faV9jRpR is 133KB 2024-03-25T19:46:25.000 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.l0faV9jRpR 2024-03-25T19:46:25.017 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/data/mon.c.tgz 2024-03-25T19:46:25.017 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-25T19:46:25.069 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.giQEXuii54 2024-03-25T19:46:25.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.giQEXuii54 2024-03-25T19:46:25.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.giQEXuii54 2024-03-25T19:46:25.243 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.giQEXuii54 is 133KB 2024-03-25T19:46:25.294 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.giQEXuii54 2024-03-25T19:46:25.307 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/data/mon.b.tgz 2024-03-25T19:46:25.307 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-25T19:46:25.325 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.6CkOorjRVk 2024-03-25T19:46:25.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6CkOorjRVk 2024-03-25T19:46:25.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.6CkOorjRVk 2024-03-25T19:46:25.517 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.6CkOorjRVk is 133KB 2024-03-25T19:46:25.569 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.6CkOorjRVk 2024-03-25T19:46:25.584 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:46:25.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:46:25.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:46:25.885 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:46:26.070 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:46:26.072 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi029/crash 2024-03-25T19:46:26.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:46:26.100 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi196/crash 2024-03-25T19:46:26.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:46:26.127 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:46:26.127 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:46:26.145 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:46:26.168 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39004.log 2024-03-25T19:46:26.168 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:46:26.219 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35306.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/valgrind/osd.4.log: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35306.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35426.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35306.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35426.log: 87.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35548.log 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35426.log.gz 2024-03-25T19:46:26.220 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35386.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35466.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35386.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35548.log.gz/var/log/ceph/ceph-client.admin.35466.log: 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35386.log.gz 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35466.log.gz 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35597.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35506.log 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35597.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35597.log.gz 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35506.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35506.log.gz 2024-03-25T19:46:26.221 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41623.log 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39004.log: /var/log/ceph/ceph-client.admin.40177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39004.log.gz 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41623.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40295.log 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39592.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41623.log.gz 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-03-25T19:46:26.222 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41194.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40295.log: /var/log/ceph/ceph-client.admin.39904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40295.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41584.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40843.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40529.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41194.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41584.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41584.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40685.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40843.log.gz 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:46:26.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40334.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/mon.c.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:46:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41818.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41818.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41818.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip/var/log/ceph/ceph-client.admin.41935.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38786.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-03-25T19:46:26.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39670.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.38786.log.gz -5 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40804.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40726.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40804.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41155.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:46:26.226 INFO:teuthology.orchestra.run.smithi029.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40726.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39709.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29229.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41272.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.29229.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29229.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35081.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41272.log: /var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41545.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41272.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41545.log: 90.2% 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35081.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-03-25T19:46:26.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39123.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41545.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39865.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz --verbose 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.39043.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40021.log: /var/log/ceph/ceph-client.admin.40099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42091.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-03-25T19:46:26.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39043.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42091.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42247.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-03-25T19:46:26.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42247.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40373.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39787.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.3.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42013.log 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39787.log.gz 2024-03-25T19:46:26.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42013.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39553.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39084.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2024-03-25T19:46:26.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39436.log: 86.6%/var/log/ceph/ceph-mgr.y.log: -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40960.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39319.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.39241.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40960.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39241.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41662.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39241.log.gz 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.41467.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36521.log 2024-03-25T19:46:26.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41467.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41662.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35489.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36521.log: /var/log/ceph/ceph-client.admin.35489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.35489.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36521.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41896.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40568.log: 92.8% 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41740.log 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42169.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-03-25T19:46:26.233 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42052.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41740.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42052.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42052.log.gzgzip 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42286.log: /var/log/ceph/ceph-client.admin.39982.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41116.log: /var/log/ceph/ceph-client.admin.41389.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2024-03-25T19:46:26.234 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40490.log: /var/log/ceph/ceph-client.admin.39163.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37528.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40607.log: /var/log/ceph/ceph-client.admin.40999.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41506.log: /var/log/ceph/ceph-client.admin.37528.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37528.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2024-03-25T19:46:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-03-25T19:46:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.41350.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2024-03-25T19:46:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39748.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-03-25T19:46:26.268 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:46:27.041 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:46:27.056 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:46:27.101 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:46:27.379 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:46:27.406 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:46:27.412 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:46:27.426 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:46:27.440 INFO:teuthology.orchestra.run.smithi029.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:46:31.028 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:46:31.364 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:46:31.366 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-25T19:46:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m5.210s 2024-03-25T19:46:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m14.583s 2024-03-25T19:46:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.543s 2024-03-25T19:46:31.618 INFO:teuthology.orchestra.run.smithi196.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:46:31.620 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-25T19:46:31.620 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m5.431s 2024-03-25T19:46:31.620 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m8.890s 2024-03-25T19:46:31.620 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.323s 2024-03-25T19:46:31.621 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:46:31.621 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi029/log 2024-03-25T19:46:31.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:46:33.935 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi196/log 2024-03-25T19:46:33.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:46:35.493 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:46:35.690 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=86187d6c7bd34a8a84f1502879cd487d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:46:35.694 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:46:35.704 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:46:35.713 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:46:35.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:46:35.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:46:35.804 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:46:35.813 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:46:35.813 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:46:35.816 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:46:35.830 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:46:35.832 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:46:35.835 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:46:35.836 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-25T19:46:35.836 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 -429us[ -429us] +/- 72ms 2024-03-25T19:46:35.836 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -314us[ -267us] +/- 59ms 2024-03-25T19:46:35.836 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +730us[ +730us] +/- 62ms 2024-03-25T19:46:35.836 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:46:35.837 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:46:35.838 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-25T19:46:35.838 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 -445us[ -445us] +/- 72ms 2024-03-25T19:46:35.838 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -324us[ -252us] +/- 59ms 2024-03-25T19:46:35.838 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +715us[ +715us] +/- 62ms 2024-03-25T19:46:35.838 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:46:35.838 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:46:35.846 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:46:35.847 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:46:35.856 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:46:35.949 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:46:36.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:46:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:46:36.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:46:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:46:36.083 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:46:36.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:46:36.105 INFO:teuthology.task.internal:Duration was 6543.341551 seconds 2024-03-25T19:46:36.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:46:36.117 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:46:36.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:46:36.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:46:36.163 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:46:36.167 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:46:36.513 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:46:36.514 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-03-25T19:46:36.514 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:46:36.534 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-25T19:46:36.535 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:46:36.561 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:46:36.562 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:46:36.578 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:46:36.630 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:46:36.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:46:36.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:46:36.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:46:36.784 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:46:36.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:46:36.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:46:36.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:46:36.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:46:36.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:46:36.876 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-03-25T19:46:36.904 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-25T19:46:36.931 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:46:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:46:36.956 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:46:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:46:36.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:46:36.994 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:46:36.994 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi029 2024-03-25T19:46:36.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:46:37.032 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318/remote/smithi196 2024-03-25T19:46:37.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:46:37.066 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:46:37.067 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:46:37.069 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:46:37.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:46:37.126 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:46:37.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:46:37.139 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:46:37.139 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:46:37.141 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:46:37.153 INFO:teuthology.orchestra.run.smithi029.stdout: 265513 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:46 /home/ubuntu/cephtest 2024-03-25T19:46:37.171 INFO:teuthology.orchestra.run.smithi196.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:46 /home/ubuntu/cephtest 2024-03-25T19:46:37.172 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:46:37.184 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:46:37.211 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:46:37.234 DEBUG:teuthology.nuke:shortname: smithi029 2024-03-25T19:46:37.235 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:46:37.257 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:48:05.211334', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDRDJbZDpFbLIzdGv/r9q9qfwyAtOCD9AXuxprSspTsrYGvfwm7W3TqHygDf1BCqYWLQ3OzFfScuo2cPStyYEQ='} 2024-03-25T19:46:37.261 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-25T19:46:37.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:46:37.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:48:05.209645', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBTCwSZrmGLv4xlFHncf1n8NvCYQh1zPjWy/plhZ2/6WW1z/UzXC2gWa2V0i5EyKIgN62bv3j0SjgFo+YQo7vQ='} 2024-03-25T19:46:37.303 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-25T19:46:37.303 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:46:37.328 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-25T19:46:37.328 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:37.343 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-25T19:46:37.343 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:46:37.353 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-25T19:46:37.366 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-25T19:46:37.366 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:37.387 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-25T19:46:41.354 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:41.388 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:41.465 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-25T19:46:41.498 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-25T19:46:45.470 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:45.499 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:46:45.581 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-25T19:46:45.609 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-25T19:46:45.682 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-25T19:46:45.710 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-25T19:46:45.815 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6543.3415513038635 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=86187d6c7bd34a8a84f1502879cd487d status: fail success: false 2024-03-25T19:46:45.815 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:46:45.893 INFO:teuthology.run:FAIL